HadISD

source is the same for Hadisd: “HadISD”, “hadisd”, “Hadisd”, “HADISD”, “Had-isd”, “had-isd”, “HAD-ISD”, “Had-isd”

get meta load, if true, read the metadata as Dataframe

from obswx import *

# Get the metadata for the HadISD dataset
met = obswx(source='hadisd')

# Load the metadata
met.get_meta(load=True).head()

	station_id	lat	lon	elevation
0	010010-99999	70.933	-8.667	9.0
1	010014-99999	59.792	5.341	48.8
2	010030-99999	77.000	15.500	12.0
3	010070-99999	78.917	11.933	7.7
4	010080-99999	78.246	15.466	26.8

metadata is from hadisd_meta.csv (last access: 2024-04-30), originally from: https://www.metoffice.gov.uk/hadobs/hadisd/v341_202403p/station_download.html

• get_data ids_source, ‘AWS’ is using AWS S3 ids, isd

• get_data station, the station id in metadata

• get_data output, optional, the path dir of output file, e.g., “pyxw/”. By default, the file nc file will be save in current work dir.

ds = met.get_data(station="010010-99999", ids_source='hadisd')

Getting data from HadISD
Goto here to help select station:  https://junjieyu-uom.github.io/obswx/hadisd_map
Downloading data from https://www.metoffice.gov.uk/hadobs/hadisd/v341_202403p/data/hadisd.3.4.1.202403p_19310101-20240401_010010-99999.nc.gz
Extracting 010010-99999.nc.gz to 010010-99999.nc

ds

<xarray.Dataset> Size: 254MB
Dimensions:                (coordinate_length: 1, time: 278217, test: 71,
                            flagged: 19, reporting_v: 19, reporting_t: 1128,
                            reporting_2: 2)
Coordinates:
    longitude              (coordinate_length) float64 8B ...
    latitude               (coordinate_length) float64 8B ...
    elevation              (coordinate_length) float64 8B ...
  * time                   (time) datetime64[ns] 2MB 1931-01-01T06:00:00 ... ...
Dimensions without coordinates: coordinate_length, test, flagged, reporting_v,
                                reporting_t, reporting_2
Data variables: (12/27)
    station_id             |S12 12B ...
    temperatures           (time) float64 2MB ...
    dewpoints              (time) float64 2MB ...
    slp                    (time) float64 2MB ...
    stnlp                  (time) float64 2MB ...
    windspeeds             (time) float64 2MB ...
    ...                     ...
    wind_gust              (time) float64 2MB ...
    past_sigwx1            (time) float64 2MB ...
    input_station_id       (time) object 2MB ...
    quality_control_flags  (time, test) float64 158MB ...
    flagged_obs            (time, flagged) float64 42MB ...
    reporting_stats        (reporting_v, reporting_t, reporting_2) float64 343kB ...
Attributes: (12/39)
    title:                       HadISD
    institution:                 Met Office Hadley Centre, Exeter, UK
    source:                      HadISD data product
    references:                  Dunn, 2019, Met Office Hadley Centre Technic...
    creator_name:                Robert Dunn
    creator_url:                 www.metoffice.gov.uk
    ...                          ...
    station_information:         Where station is a composite the station id ...
    Conventions:                 CF-1.6
    Metadata_Conventions:        Unidata Dataset Discovery v1.0, CF Discrete ...
    featureType:                 timeSeries
    processing_date:             04-Apr-2024
    history:                     Created by mk_netcdf_files.py \nDuplicate Mo...

xarray.Dataset

• Dimensions:
  • coordinate_length: 1
  • time: 278217
  • test: 71
  • flagged: 19
  • reporting_v: 19
  • reporting_t: 1128
  • reporting_2: 2
• Coordinates: (4)
  • longitude
    (coordinate_length)
    float64
    ...

    standard_name :

    longitude

    long_name :

    station longitude

    units :

    degrees_east

    axis :

    X

    [1 values with dtype=float64]

  • latitude
    (coordinate_length)
    float64
    ...

    standard_name :

    latitude

    long_name :

    station latitude

    units :

    degrees_north

    axis :

    Y

    [1 values with dtype=float64]

  • elevation
    (coordinate_length)
    float64
    ...

    standard_name :

    surface_altitude

    long_name :

    vertical distance above the surface

    units :

    meters

    axis :

    Z

    positive :

    up

    [1 values with dtype=float64]

  • time
    (time)
    datetime64[ns]
    1931-01-01T06:00:00 ... 2024-03-...

    long_name :

    time_of_measurement

    valid_min :

    0.0

    standard_name :

    time

    array(['1931-01-01T06:00:00.000000000', '1931-01-01T12:00:00.000000000',
           '1931-01-01T18:00:00.000000000', ..., '2024-03-27T15:00:00.000000000',
           '2024-03-27T16:00:00.000000000', '2024-03-27T17:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (27)
  • station_id
    ()
    |S12
    ...

    long_name :

    Station ID number

    [1 values with dtype=|S12]

  • temperatures
    (time)
    float64
    ...

    long_name :

    Dry bulb air temperature at screen height (~2m)

    units :

    degree_Celsius

    flagged_value :

    -2e+30

    valid_min :

    -26.0

    valid_max :

    37.2

    standard_name :

    surface_temperature

    cell_methods :

    latitude: longitude: time: point (nearest to reporting hour)

    [278217 values with dtype=float64]

  • dewpoints
    (time)
    float64
    ...

    long_name :

    Dew point temperature at screen height (~2m)

    units :

    degree_Celsius

    flagged_value :

    -2e+30

    valid_min :

    -67.6

    valid_max :

    14.6

    standard_name :

    dew_point_temperature

    cell_methods :

    latitude: longitude: time: point (nearest to reporting hour)

    [278217 values with dtype=float64]

  • slp
    (time)
    float64
    ...

    long_name :

    Reported Sea Level Pressure at screen height (~2m)

    units :

    hPa

    flagged_value :

    -2e+30

    valid_min :

    942.0

    valid_max :

    1071.3

    standard_name :

    air_pressure_at_sea_level

    cell_methods :

    latitude: longitude: time: point (nearest to reporting hour)

    [278217 values with dtype=float64]

  • stnlp
    (time)
    float64
    ...

    long_name :

    Reported Station Level Pressure at screen height (~2m)

    units :

    hPa

    flagged_value :

    -2e+30

    valid_min :

    942.0

    valid_max :

    1071.3

    standard_name :

    surface_air_pressure

    cell_methods :

    latitude: longitude: time: point (nearest to reporting hour)

    [278217 values with dtype=float64]

  • windspeeds
    (time)
    float64
    ...

    long_name :

    Wind speed at mast height (~10m)

    units :

    meters per second

    flagged_value :

    -2e+30

    valid_min :

    0.0

    valid_max :

    38.1

    standard_name :

    wind_speed

    cell_methods :

    latitude: longitude: time: point (nearest to reporting hour)

    [278217 values with dtype=float64]

  • winddirs
    (time)
    float64
    ...

    long_name :

    Wind Direction at mast height (~10m)

    units :

    degree

    flagged_value :

    -888

    valid_min :

    0

    valid_max :

    360

    standard_name :

    wind_from_direction

    cell_methods :

    latitude: longitude: time: point (nearest to reporting hour)

    [278217 values with dtype=float64]

  • total_cloud_cover
    (time)
    float64
    ...

    long_name :

    Total cloud cover (oktas)

    units :

    1

    flagged_value :

    -888

    valid_min :

    0

    valid_max :

    8

    standard_name :

    cloud_area_fraction

    cell_methods :

    latitude: longitude: time: point (derived in priority order GA, GF, GD - see ISD documentation, nearest to reporting hour)

    [278217 values with dtype=float64]

  • low_cloud_cover
    (time)
    float64
    ...

    long_name :

    Low cloud cover (oktas)

    units :

    1

    flagged_value :

    -888

    valid_min :

    0

    valid_max :

    8

    standard_name :

    low_type_cloud_area_fraction

    cell_methods :

    latitude: longitude: time: point (derived in priority order GA, GF, GD - see ISD documentation, nearest to reporting hour)

    [278217 values with dtype=float64]

  • mid_cloud_cover
    (time)
    float64
    ...

    long_name :

    Mid cloud cover (oktas)

    units :

    1

    flagged_value :

    -888

    valid_min :

    0

    valid_max :

    8

    standard_name :

    medium_type_cloud_area_fraction

    cell_methods :

    latitude: longitude: time: point (derived in priority order GA, GF, GD - see ISD documentation, nearest to reporting hour)

    [278217 values with dtype=float64]

  • high_cloud_cover
    (time)
    float64
    ...

    long_name :

    High cloud cover (oktas)

    units :

    1

    flagged_value :

    -888

    valid_min :

    0

    valid_max :

    8

    standard_name :

    high_type_cloud_area_fraction

    cell_methods :

    latitude: longitude: time: point (derived in priority order GA, GF, GD - see ISD documentation, nearest to reporting hour)

    [278217 values with dtype=float64]

  • precip1_depth
    (time)
    float64
    ...

    long_name :

    Depth of Precipitation Reported in 1 hour (from all four ISD fields)

    units :

    mm

    flagged_value :

    -2e+30

    valid_min :

    0.0

    valid_max :

    999.8

    standard_name :

    lwe_thickness_of_precipitation_amount

    cell_methods :

    latitude: longitude: time: sum

    [278217 values with dtype=float64]

  • precip2_depth
    (time)
    float64
    ...

    long_name :

    Depth of Precipitation Reported in 2 hour (from all four ISD fields)

    units :

    mm

    flagged_value :

    -2e+30

    valid_min :

    0.0

    valid_max :

    999.8

    standard_name :

    lwe_thickness_of_precipitation_amount

    cell_methods :

    latitude: longitude: time: sum

    [278217 values with dtype=float64]

  • precip3_depth
    (time)
    float64
    ...

    long_name :

    Depth of Precipitation Reported in 3 hours (from all four ISD fields)

    units :

    mm

    flagged_value :

    -2e+30

    valid_min :

    0.0

    valid_max :

    999.8

    standard_name :

    lwe_thickness_of_precipitation_amount

    cell_methods :

    latitude: longitude: time: sum

    [278217 values with dtype=float64]

  • precip6_depth
    (time)
    float64
    ...

    long_name :

    Depth of Precipitation Reported in 6 hour (from all four ISD fields)

    units :

    mm

    flagged_value :

    -2e+30

    valid_min :

    0.0

    valid_max :

    999.8

    standard_name :

    lwe_thickness_of_precipitation_amount

    cell_methods :

    latitude: longitude: time: sum

    [278217 values with dtype=float64]

  • precip9_depth
    (time)
    float64
    ...

    long_name :

    Depth of Precipitation Reported in 9 hour (from all four ISD fields)

    units :

    mm

    flagged_value :

    -2e+30

    valid_min :

    0.0

    valid_max :

    999.8

    standard_name :

    lwe_thickness_of_precipitation_amount

    cell_methods :

    latitude: longitude: time: sum

    [278217 values with dtype=float64]

  • precip12_depth
    (time)
    float64
    ...

    long_name :

    Depth of Precipitation Reported in 12 hour (from all four ISD fields)

    units :

    mm

    flagged_value :

    -2e+30

    valid_min :

    0.0

    valid_max :

    999.8

    standard_name :

    lwe_thickness_of_precipitation_amount

    cell_methods :

    latitude: longitude: time: sum

    [278217 values with dtype=float64]

  • precip15_depth
    (time)
    float64
    ...

    long_name :

    Depth of Precipitation Reported in 15 hour (from all four ISD fields)

    units :

    mm

    flagged_value :

    -2e+30

    valid_min :

    0.0

    valid_max :

    999.8

    standard_name :

    lwe_thickness_of_precipitation_amount

    cell_methods :

    latitude: longitude: time: sum

    [278217 values with dtype=float64]

  • precip18_depth
    (time)
    float64
    ...

    long_name :

    Depth of Precipitation Reported in 18 hour (from all four ISD fields)

    units :

    mm

    flagged_value :

    -2e+30

    valid_min :

    0.0

    valid_max :

    999.8

    standard_name :

    lwe_thickness_of_precipitation_amount

    cell_methods :

    latitude: longitude: time: sum

    [278217 values with dtype=float64]

  • precip24_depth
    (time)
    float64
    ...

    long_name :

    Depth of Precipitation Reported in 24 hour (from all four ISD fields)

    units :

    mm

    flagged_value :

    -2e+30

    valid_min :

    0.0

    valid_max :

    999.8

    standard_name :

    lwe_thickness_of_precipitation_amount

    cell_methods :

    latitude: longitude: time: sum

    [278217 values with dtype=float64]

  • cloud_base
    (time)
    float64
    ...

    long_name :

    Cloud base of lowest cloud layer

    units :

    meters

    valid_min :

    0.0

    valid_max :

    22000.0

    standard_name :

    cloud_base_altitude

    cell_methods :

    latitude: longitude: time: point (nearest to reporting hour)

    [278217 values with dtype=float64]

  • wind_gust
    (time)
    float64
    ...

    long_name :

    Wind Gust Speed at mast height (~10m)

    units :

    meters per second

    valid_min :

    0.0

    valid_max :

    200.0

    standard_name :

    wind_speed_of_gust

    cell_methods :

    latitude: longitude: time: point (nearest to reporting hour)

    [278217 values with dtype=float64]

  • past_sigwx1
    (time)
    float64
    ...

    long_name :

    Reported past significant weather phenomena

    units :

    1

    valid_min :

    0

    valid_max :

    9

    cell_methods :

    latitude: longitude: point (interval: 1 day)

    [278217 values with dtype=float64]

  • input_station_id
    (time)
    object
    ...

    long_name :

    Primary source for timestep (may be multiple sources for composite stations). USAF-WBAN from ISD source

    [278217 values with dtype=object]

  • quality_control_flags
    (time, test)
    float64
    ...

    units :

    1

    long_name :

    Quality Control status for individual obs

    [19753407 values with dtype=float64]

  • flagged_obs
    (time, flagged)
    float64
    ...

    units :

    1

    long_name :

    Observation Values removed by QC flags temperatures dewpoints slp stnlp windspeeds winddirs total_cloud_cover low_cloud_cover mid_cloud_cover high_cloud_cover precip1_depth precip2_depth precip3_depth precip6_depth precip9_depth precip12_depth precip15_depth precip18_depth precip24_depth

    cell_methods :

    latitude: longitude: time: point

    [5286123 values with dtype=float64]

  • reporting_stats
    (reporting_v, reporting_t, reporting_2)
    float64
    ...

    units :

    1

    long_name :

    Reporting frequency and accuracy for each month and variable in following order: temperatures dewpoints slp stnlp windspeeds winddirs total_cloud_cover low_cloud_cover mid_cloud_cover high_cloud_cover precip1_depth precip2_depth precip3_depth precip6_depth precip9_depth precip12_depth precip15_depth precip18_depth precip24_depth

    [42864 values with dtype=float64]

• Indexes: (1)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['1931-01-01 06:00:00', '1931-01-01 12:00:00',
                   '1931-01-01 18:00:00', '1931-01-02 06:00:00',
                   '1931-01-02 12:00:00', '1931-01-02 18:00:00',
                   '1931-01-03 06:00:00', '1931-01-03 12:00:00',
                   '1931-01-03 18:00:00', '1931-01-04 06:00:00',
                   ...
                   '2024-03-27 08:00:00', '2024-03-27 09:00:00',
                   '2024-03-27 10:00:00', '2024-03-27 11:00:00',
                   '2024-03-27 12:00:00', '2024-03-27 13:00:00',
                   '2024-03-27 14:00:00', '2024-03-27 15:00:00',
                   '2024-03-27 16:00:00', '2024-03-27 17:00:00'],
                  dtype='datetime64[ns]', name='time', length=278217, freq=None))

• Attributes: (39)

  title :

  HadISD

  institution :

  Met Office Hadley Centre, Exeter, UK

  source :

  HadISD data product

  references :

  Dunn, 2019, Met Office Hadley Centre Technical Note #103; Dunn, Willett, Parker & Mitchell, 2016, Geosci. Instrum. Method. Data Syst., 5, 473-491

  creator_name :

  Robert Dunn

  creator_url :

  www.metoffice.gov.uk

  creator_email :

  robert.dunn@metoffice.gov.uk

  keywords :

  sub-daily, station, extremes, temperature, dewpoint, pressure, wind_speed, wind_direction, cloud_cover

  summary :

  Quality-controlled, sub-daily, station dataset containing temperature, dewpoint, pressure, wind speed & direction and cloud cover.

  hadisd_version :

  3.4.1.202403p

  isd_version :

  ---

  isd_date_full :

  5 February 2024

  isd_date_update :

  3 April 2024

  licence :

  HadISD is distributed under the Non Commercial Government Licence: http://www.nationalarchives.gov.uk/doc/non-commercial-government-licence/non-commercial-government-licence.htm. The data are available for non-commercial use with attribution to the data providers. Please cite Dunn et al (2012), Dunn et al (2014), Dunn et al (2016) and Smith et al (2011) and state the version used. This product may contain data which are governed by WMO Policy following WMO Resolution 40 Annex 1 alongside additional data that may have restrictions placed on their commercial use by the data owners. Any redistribution of this product should be accompanied by a similar statement of usage policy.

  processing_level :

  Quality controlled timeseries at station level

  hadisd_time_coverage_start :

  1931-01-01T00:00Z

  hadisd_time_coverage_end :

  2024-04-01T00:00Z

  cdm_data_type :

  station

  acknowledgement :

  RJHD was supported by the Joint BEIS/Defra Met Office Hadley Centre Climate Programme (GA01101).

  geospatial_lat_max :

  70.933

  geospatial_lat_min :

  70.933

  geospatial_lat_units :

  degrees

  geospatial_lon_max :

  -8.667

  geospatial_lon_min :

  -8.667

  geospatial_lon_units :

  degrees

  geospatial_vertical_max :

  9.0

  geospatial_vertical_min :

  9.0

  geospatial_vertical_units :

  m

  station_id :

  010010-99999

  time_coverage_start :

  1931-01-01T06:00Z

  time_coverage_end :

  2024-03-27T17:00Z

  date_created :

  2024-04-04, 00:04

  qc_code_version :

  279

  station_information :

  Where station is a composite the station id refers to the primary source used in the timestep and does apply to all elements

  Conventions :

  CF-1.6

  Metadata_Conventions :

  Unidata Dataset Discovery v1.0, CF Discrete Sampling Geometries Conventions

  featureType :

  timeSeries

  processing_date :

  04-Apr-2024

  history :

  Created by mk_netcdf_files.py Duplicate Months Check 2024-04-03, 16:52 Isolated Odd Cluster Check 2024-04-03, 16:53 Frequent Values Check 2024-04-03, 16:53 Distributional Gap Check 2024-04-03, 16:53 World Record Check 2024-04-03, 16:53 Streak Check 2024-04-03, 16:53 Climatological Check 2024-04-03, 16:53 Spike Check 2024-04-03, 16:54 Temperature-Humidity Cross Check 2024-04-03, 16:54 Cloud - Logical Cross Check 2024-04-03, 16:54 Excess Variance Check 2024-04-03, 16:54 Pressure Cross Check 2024-04-03, 16:54 Precipitation Cross Check 2024-04-03, 16:54 Neighbour Outlier Check 2024-04-04, 00:04 Clean Up Months 2024-04-04, 00:04 Masking 2024-04-04, 00:04

transform to pandas dataframe

ds['temperatures'].to_dataframe()

	temperatures
time
1931-01-01 06:00:00	-5.0
1931-01-01 12:00:00	-5.0
1931-01-01 18:00:00	-5.0
1931-01-02 06:00:00	-5.0
1931-01-02 12:00:00	-5.0
...	...
2024-03-27 13:00:00	-2.7
2024-03-27 14:00:00	-2.7
2024-03-27 15:00:00	-2.7
2024-03-27 16:00:00	-3.0
2024-03-27 17:00:00	-3.5

278217 rows × 1 columns

save to a csv file

df = ds['temperatures'].to_dataframe()
df.to_csv('data.csv')