European Users

[Dave Exton]

Phil and Alan,

I've finally produced a result! I have no idea if I'm getting windspeeds in M/S or otherwise, but I'll sort that out when I've recovered from actually seeing the stations plot.

All I have to do now is remember what I had to do to avhieve a result.

Again, thank you. maybe now I can get some sleep.

[Dave Exton]

A BIG thank you to Phil Layton and Alan Duckers for helping me to solve the Missing Synops in Europe problem! Their efforts were invaluable, and have completely restored my faith in the wonderful Digital Atmosphere Program.

Bassed on their instructions to me, and addapted slightly, to suit my own circumstances, I have listed all the steps needed to plot a full synoptic map of Europe.

Please feel free to point out any errors so that I can correct them.

To add missing SYNOP data to European countries in DA
Download data from your favourite source about 1.5 hrs after the SYNOP time of interest.
Plot the data.
When plotting this data, there may be gaps in European countries.
Open OGIMET
Select ‘SYNOP BY COUNTRIES’
Select from DROPDOWN: country of interest
Ensure ‘FORMAT’ is set to TXT
Select TIME INTERVAL to date of interest (DROPDOWN: Year, Month, Day, Hour)
List of SYNOPS is produced. PRESS ‘Select All’, then ‘Copy’
Open NOTE PAD
Choose ‘PASTE
All SYNOPS are now ready to be edited in NOTE PAD
The next step involves removing all TIME STAMPS in the SYNOP reports (within NOTE PAD)
Go to EDIT,
SELECT ‘Replace’
COPY a time stamp from the SYNOP reports
PASTE into ‘Replace’
Leave the ‘With what’ box empty (since you are removing all the TIME STAMPS and not replacing them with anything)
An amended SYNOP report is produced without TIME STAMPS
Now Type in a TIME STAMP e.g AAXX 0461 (From the list of time stamps you have removed)
And SAVE AS the document to the desk top and I name it with with an abbreviated country name and time. You now have a NOTE PAD file on your DESK TOP.
Repeat the above steps for any countries that are missing from the DA download.
I also have a SHORT CUT to the DA Data File, so that I can open the data file directly.
The location of the data file can be found from the HELP section / about, from the DA top menu bar.
I then copy the NOTE PAD file and PASTE into the data file.
In order for this to work, the file header needs to be correct (as above e.g. AAXX 0461)
IN DIGITAL ATMOSPHERE, select FILE from the menu, and the select IMPORT DIRECTORY.
When all of the directory has been imported, use the station plot command.
And as if by magic, all he stations should plot.

[alanduckers]

Hi Dave,
good to hear you have it working now.
All we need now is for Tim to carry out a large update so you can input BUFR data from the sources on the internet into DA that would extremly helpful
cheers,
Alan

[Dave Exton]

Yes Indeed! Fingers crossed

I'm just now starting to worry about the m/s windspeed issue. Certainly, all my plots (Germany, Norway, Sweden and Latvia), look okay, but I'm now not so sure.

Anyway, Thanks again for your help.

I'm not sure if you'd be interested, but some time back I made a Word File of all the UK Shipping Forcast area (just the demarkation lines, without any text). I always add them in to my Custom Geography, which I transpose on to all my European maps. Let me know if you'd like a copy and I'll see what I can do.

Regs, Dave.

[Tim Vasquez]

We looked at fixing BUFR and the problem is the specification is enormously (ridiculously) complicated, so it can't be built natively into Digital Atmosphere. Also when I started looking around I was having trouble finding the BUFR data itself, since it is binary. So not many datastreams choose to carry any of the data. Not even the TGFTP server is populating the data... you can look around in here https://tgftp.nws.noaa.gov/data/raw/is/ and in other directories but it is either missing or outdated. I talked to them and they said they weren't going to fix it.

It is possible some of the data may exist on the NCEP production servers at https://www.ftp.ncep.noaa.gov/data/nccf/ but I am not sure where it would be.

The solution for this is going to involve:

1. Figuring out who out there is capturing BUFR and placing it online, and

2. Using an off-the-shelf BUFR converter to create CSV or JSON fields and then writing some code on our end to build SYNOP and upper air TEMP back up from that, and then offer it for download.

Some of you may be able to do both... if so I can point the data retrieval to your server. Otherwise I will try to address this when we can return to the BUFR issues.

[Dave Exton]

Thanks Tim,
Thanks to the good people on this forum, I have learned of an easy workaround to fix the problem.
Details are in this string.

[ianp]

Thanks for the pointer folks.
I've cobbled together a python script to edit the ogimet output
I hope it works for you
I've done nothing clever to select the data, if the ogimet txt file format changes, some tweaks may be required.

# file to read ogimet SYNOP format files and write a SYNOP ascii file
# to read into Digital Atmosphere
#
from datetime import datetime
from pathlib import Path
import os

def readInteger(inputPrompt,low,high):
"""read integer within specific bounds"""
while True:
try:
number = int(input(inputPrompt))
except ValueError:
print('oops not an integer')
continue
if number >= low and number <= high:
return number
else:
print('try again, number out of range')


homeDir = Path('D:/meteorology_data/METEO/')

# check current time
now = datetime.now()

# select date and time to read ogimet txt files
# for specific countries
# typically Germany, Hungary, Portugal and Sweden in Europe
# go to https://www.ogimet.com/synopsc.phtml.en
# set begin and end at same time if you like
# select the country and click send
# then save page as ...
# the file in "YYMMDDHH * ogimet * .txt" format
# I tend to use a previous file and change a few numbers
#


# a couple of lines to save typing in the year every time
if now.month<3:
useYear = readInteger("Please enter year ", now.year - 1, now.year)
else:
useYear = now.year

# enter the month, day and hour to process
useMonth = readInteger("Please enter month ", 1, 12)
# beware I don't check for days that aren't in a month
useDay = readInteger("Please enter day ", 1, 31)
useHour = readInteger("please enter hour ", 0, 23)

readTime = datetime(useYear, useMonth, useDay, useHour, 0, 0)

# create a time string to match the ogimet file date time
timeString = readTime.strftime("%Y%m%d%H%M")
# print("date time to use: ", timeString)
# create a glob string to search for files with the date format
globStr = readTime.strftime("%y%m%d%H")
# print("date time glob str:", globStr)

# I have a folder for each current month
monthFolder = Path(readTime.strftime("%Y-%m"))

# modify the path to the month folder for output
homeDir = homeDir / monthFolder

# select files of interest
fileList = []
fileList = list(homeDir.glob(globStr+'*ogimet*.txt'))


for k, workingFile in enumerate(fileList):
filePathStr = str(fileList[k])
outFileStr = filePathStr.replace(".txt",".asc")
print()
print("output file name", outFileStr)


# read file line-by-line and find AAXX with the timeString
# or lines starting with 333
ogimetFile = open(fileList[k])
with open(fileList[k]) as ogimetFile:
ogimetData = ogimetFile.readlines()
print('input file lines ',len(ogimetData))

# process the data
# skip header rows and find first line of SYNOP data
j=0
while "AAXX" not in ogimetData[j] and j in range(len(ogimetData)):
j=j+1

# open output file
outFile = open(outFileStr, "w")
# write each valid line
for i in range(j, len(ogimetData)):
if ogimetData[0] == "\n" : # only a new line character
# blank line
print()
outFile.write("\n")
elif ogimetData[0] == "#":
# comment - do nothing
pass
elif "AAXX" in ogimetData and ogimetData[:12] == timeString:
# first line of data
print(ogimetData[13:23])
outFile.write(ogimetData[13:23]+"\n")
print(ogimetData[24:len(ogimetData)],end='') # new line already there
outFile.write(ogimetData[24:len(ogimetData)])
elif " 333 " in ogimetData[i] and ogimetData[i-1][:12] == timeString:
# second line SYNOP section three
print(ogimetData[i].lstrip(),end='') # new line already there
outFile.write(ogimetData[i].lstrip())
elif " 333 " in ogimetData[i-1] and ogimetData[i-2][:12] == timeString:
# the occasional third line
print(ogimetData[i].lstrip(),end='') # new line already there
outFile.write(ogimetData[i].lstrip())
# else:
# print("something odd happened")

outFile.close()

print("end of processing")

[ianp]

sorry the indents have disappeared