Domestic Electrical Load Metering, Hourly Data 1994-2014
South Africa, 1994 - 2014
Get Microdata
Reference ID
zaf-erc-delmh-1994-2014-v1
Producer(s)
Wiebke Toussaint
Collections
Created on
May 29, 2019
Last modified
Apr 15, 2020
Page views
9269
Downloads
1627
Identification
Survey ID Number
zaf-erc-delmh-1994-2014-v1
Title
Domestic Electrical Load Metering, Hourly Data 1994-2014
Subtitle
Hourly Data 1994-2014
Country
| Name | Country code |
|---|---|
| South Africa | ZAF |
Study type
Service Provision Assessment [hh/spa]
Abstract
This data is an aggregated subset of the 5-minute interval electricity metering data available in the Domestic Electrical Load Metering Data (DELM) 1994-2014 available in DataFirst's secure centre. The large volume and high metering cadence of the DELM 1994-2014 data is unwieldy to access and process. Many applications that do not require the granularity of the DELM 1994-2014 data will be able to extract value more effectively and conveniently from aggregate values. This dataset contains all current (Amps) observations aggregated to hourly values. It can be easily merged with the Domestic Electrical Load Survey Key Variables 1994-2014 data to link socio-demographic varibles with household consumption data. This dataset and similar custom datasets can be produced from the DELM 1994-2014 dataset with the python package delprocess. The data processing section includes a description of how this dataset was created. The development of the tools to create this dataset was funded by the South African National Energy Development Initiative (SANEDI).
Kind of Data
Observation data
Unit of Analysis
Households
Version
Version Description
v1: Edited, anonymised data for licensed distribution
Version Date
2019-05-23
Scope
Notes
The scope of the hourly South African Domestic Electrical Load Metering data includes:
CURRENT: Amperes (A) aggregated over a 60 minute interval defined to start daily at 00:00:00 - 00:59:59.
CURRENT: Amperes (A) aggregated over a 60 minute interval defined to start daily at 00:00:00 - 00:59:59.
Keywords
| Keyword |
|---|
| current |
| South Africa |
| rural |
| electrification |
| load data |
| time-of-use |
| electrical |
| household |
| consumption |
| energy |
| electricity |
| domestic |
| residential |
| behaviour |
Coverage
Geographic Coverage
The study had national coverage.
Geographic Unit
The DELMH 1994-2014 dataset does not contain geographic information. When combined with the DELSKV 1994-2014 dataset, the lowest unit of geographic aggregation is settlement/suburb. Municipal-level data is also available in the DELSKV data which can be merged with the DELMH data.
Universe
The metering study covers electrified households that received electricity either directly from Eskom or from their local municipality. Particular attention was devoted to rural and low income households, as well as surveying households electrified over a range of years, thus having had access to electricity from recent times to several decades.
Producers and sponsors
Primary investigators
| Name | Affiliation |
|---|---|
| Wiebke Toussaint | University of Cape Town |
Funding Agency/Sponsor
| Name | Abbreviation | Role |
|---|---|---|
| South African National Energy Development Initiative | SANEDI | Funder |
Other Identifications/Acknowledgments
| Name | Affiliation | Role |
|---|---|---|
| Marcus Dekenah | MDekenah Consulting | [Expert knowledge for] understanding database design and conveying details around data collection |
Data Collection
Dates of Data Collection
| Start | End |
|---|---|
| 1994 | 2014 |
Data Collection Mode
Other [oth]
Data Processing
Data Editing
This data has been produced by aggregating all current (Amps) metering data from the DELMS 1994-2014 dataset using the reduceRawProfiles function from the delprocess python package (https://github.com/wiebket/delprocess: release v1.0). Full instructions on how to use delprocess to aggregate metering data are in the README file contained in the package.
INVALID READINGS
The 'Valid' indicator of readings was converted to 1 (valid) and 0 (invalid). Missing 'Valid' indicators were filled with 0 values.
MISSING VALUES
Missing readings were treated as per pandas.dataframe.mean default: skipna = True; i.e. missing values are excluded when computing results.
DATA AGGREGATION (OBSERVATIONS)
The following processing steps were performed to produce the aggregate dataset:
0. 'Datefield' values were converted to integer values, rounded to 9 positions left of the decimal, and converted to a numpy datetime64 object with nano-second units. This was done to coerce the data to consistent time intervals.
1. readings grouped by RecorderID and ProfileID
2. grouped data resampled to hourly values ('Datefield' column converted to 'H' offset)
3. mean meter reading value and 'Valid' indicator calculated over resampled intervals
4. rows with all missing values removed
5. aggregated 'Valid' indicator set to 0 unless it was 1 (i.e. if one reading was marked as invalid, the mean 'Valid' indicator would be less than 1 and the aggregate 'Valid' indicator was set to 0, thus marking the aggregated validity as invalid)
DATA AGGREGATION (STUDY CYCLES)
Data was aggregated per year, across temporally overlapping study cycles.
INVALID READINGS
The 'Valid' indicator of readings was converted to 1 (valid) and 0 (invalid). Missing 'Valid' indicators were filled with 0 values.
MISSING VALUES
Missing readings were treated as per pandas.dataframe.mean default: skipna = True; i.e. missing values are excluded when computing results.
DATA AGGREGATION (OBSERVATIONS)
The following processing steps were performed to produce the aggregate dataset:
0. 'Datefield' values were converted to integer values, rounded to 9 positions left of the decimal, and converted to a numpy datetime64 object with nano-second units. This was done to coerce the data to consistent time intervals.
1. readings grouped by RecorderID and ProfileID
2. grouped data resampled to hourly values ('Datefield' column converted to 'H' offset)
3. mean meter reading value and 'Valid' indicator calculated over resampled intervals
4. rows with all missing values removed
5. aggregated 'Valid' indicator set to 0 unless it was 1 (i.e. if one reading was marked as invalid, the mean 'Valid' indicator would be less than 1 and the aggregate 'Valid' indicator was set to 0, thus marking the aggregated validity as invalid)
DATA AGGREGATION (STUDY CYCLES)
Data was aggregated per year, across temporally overlapping study cycles.
Access policy
Contacts
| Name | Affiliation | URL | |
|---|---|---|---|
| DataFirst Support | University of Cape Town | support@data1st.org | support.data1st.org |
Access conditions
Licensed use files, available with restrictions
Citation requirements
Toussaint, Wiebke. Domestic Electrical Load Metering, Hourly Data 1994-2014 [dataset]. Version 1. Johannesburg: SANEDI [funders]. Cape Town: Energy Research Centre, UCT [producers], 2014. Cape Town: DataFirst [distributor], 2019. DOI: https://doi.org/10.25828/56nh-fw77
Access authority
| Name | Affiliation | URL | |
|---|---|---|---|
| DataFirst | University of Cape Town | support@data1st.org | support.data1st.org |
Metadata production
Producers
| Name | Affiliation | Role |
|---|---|---|
| DataFirst | University of Cape Town | Metadata producer |
Date of Metadata Production
2020-04-15
DDI Document version
Version 4