Combination of techniques for online price data collection: Case of Ukraine

National bank of Ukraine

Geneva, April 2023

National bank of Ukraine

Summary

2

Online data collection project was launched in 2015. Initially, it was a set of scripts for price scrapping from online sources with semi- manual data processing in Excel after. Data was collected mainly from the websites of small regional supermarkets

Update of online data collection system to a fully automated data collection/filtering/calculation system was started in 2018. From the very beginning, it was planned to receive information about food products and fuel prices, as well as certain services with regional breakdown, prices of which are available online. 5 large supermarkets were selected as the main source of information

The project also included sentiment analysis module development - collection of text information, processing, classification and determination of its sentiments using a CNN (confluence neural network) approach

Start

Development

Additional initiatives

3

What are the goals of the project?

The goal of the project was to launch automated system of data collection, error correction, and price index calculation to:

• obtain recent data in uncertainty times and high inflation rates for consideration at the Monetary Policy Committee meetings

• track the effects of monetary transmission

• conduct research (on "price stickiness“, regional price development and other microeconomic facts of pricing)

Architecture of the model has to be expandable for real estate prices, employment and output estimation

4

Online data collection system architecture

- : by dictionary (barcodes of 20K unique products out of 65K)

- of missing data and technical website errors

- of outliers

Indexes aggregation (128 types)

- database access (data and visualization) - methodological clarifications - feedback

- to adjust filter parameters

- basket structure and weights

Web-Scrapping

- 5 largest supermarkets (>70% of retail turnover) and fuel stations

- high-tech Python code (including dynamic HTML scraping module)

- daily basis

Econometric models

- dependence on the exchange rate and MTP inflation (daily approximation)

- services:

- dependence on seasonality, fuel, administrative price setting

Raw data and proxies

for non-foods:

for services:

Intellectual backtests

Regular review

Update

Core (Oracle project)

Users

Filtration

Correction

Trimming

5

Coverage of CPI basket by web-scraped items is more than 70%

Core CPI

Regulated prices and tariffs

Raw food

Fuel

* Shaded field is the share of products for which data are not collected. Source: State Statistics Committee, NBU calculations.

Items scrapped: appr. 65 000 Items selected for basket: appr. 20 000 Groups: 128

Risks for online data collection system

Change of supermarket’s site structure

Risk Probability

Low

Average

High

6

Model needs an update. Can take 1-2 weeks

Comment

New site platforms

Need to update the set of products and price filter settings every year

Update of dictionary can take two weeks or more

Model needs an update. Can take a week

Non-food products prices are difficult to collect (goods are changed frequently)

Update of econometric models for non-food prices estimation. More frequent update of a dictionary

7

The difference of scrapped data with official data widened as the situation in Ukraine became unstable

-3,0

-2,0

-1,0

0,0

1,0

2,0

3,0

4,0

5,0

01 .2

1

02 .2

1

03 .2

1

04 .2

1

05 .2

1

06 .2

1

07 .2

1

08 .2

1

09 .2

1

10 .2

1

11 .2

1

12 .2

1

01 .2

2

02 .2

2

03 .2

2

04 .2

2

05 .2

2

06 .2

2

07 .2

2

08 .2

2

09 .2

2

10 .2

2

11 .2

2

12 .2

2

01 .2

3

02 .2

3

CPI data comparision, pp

CPI CPI (realtime) Difference

-2,0

-1,0

0,0

1,0

2,0

3,0

4,0

5,0

01 .2

1

02 .2

1

03 .2

1

04 .2

1

05 .2

1

06 .2

1

07 .2

1

08 .2

1

09 .2

1

10 .2

1

11 .2

1

12 .2

1

01 .2

2

02 .2

2

03 .2

2

04 .2

2

05 .2

2

06 .2

2

07 .2

2

08 .2

2

09 .2

2

10 .2

2

11 .2

2

12 .2

2

01 .2

3

02 .2

3

Core CPI data comparision, pp

Core Core (realtime) Difference

From March, 2022 prices fluctuations become more unpredictable. All risks were realized.

The higher inflation – the bigger difference between data collected in real-time mode and official data

13.06.2023

1

Inventory of assessments of the environmental damage resulting from the Russian invasion of Ukraine Disclaimer: The inter-agency group is not responsible for the accuracy of the contents of the studies in this collection. The titles of the studies and the findings, interpretations and conclusions expressed therein are those of the authors and do not necessarily reflect the views of the United Nations or its officials or Member States. Mention of a commercial company, product or service does not imply endorsement by the United Nations or its Member States.

Title Organizations involved Content The Environmental

Impact of the Ukraine Conflict – A Preliminary

Review 14 October 2022

UNEP & GRID Arendal

The report uses reviews of official documentation on environmental and related human health impacts from past conflicts; information from the Government of Ukraine; limited remote sensing information; and non-verified reports from social media posts and regular media for framing issues, the approach and assessment. The summarized issues and impacts include chemical industries and chemicals associated with armed conflict; fuel and associated infrastructure; waste and waste infrastructure; urban and critical infrastructure; and damage to agriculture and to nature.

More information can be found here. Ukraine Rapid Damage and Needs Assessment

August 2022 Follow-up underway

World Bank & Government of Ukraine &

European Commission

The Rapid Damage and Needs Assessment is a part of an ongoing effort to gather information on Ukraine’s damage of infrastructure and losses. It also assesses the scale of economic and social needs for Ukraine currently as well as post-conflict towards recovery and reconstruction. It is not limited to environmental issues. A follow-up study is underway, with UN involvement.

More information can be found here More information on water & energy security analysis by hydrosolutions GmbH can be found

here.

13.06.2023

2

Title Organizations involved Content Ukraine Rapid Damage

and Needs Assessment 2 March 2023

World Bank Group & Government of Ukraine &

European Union Services & United Nations

As a follow up to the first Rapid Damage and Needs Assessment this assessment takes stock of Ukraine’s damage and losses from one year of war. The report identifies key damages to a range of affected of sectors and implies recovery costs, foundations for strategy, and reconstruction perspectives. Sectors covered in the report include, but are not limited to, human impact, the productive sector, the infrastructure sector, environmental areas, natural resources and forestry areas.

More information can be found here. Environmental Impacts of the war in Ukraine

and prospects for a green reconstruction

1 July 2022

OECD This policy brief reviews the impacts of the Russian invasion of Ukraine on the environment and environmental infrastructure in Ukraine. It suggests methods for a green reconstruction towards a net-zero economy.

More information can be found here.

Ecodozor Live

Zoï Environment Network (Switzerland) &

OSCE Project Coordinator in Ukraine &

UNEP

Ecodozor is a tool used to map the environmental consequences and risks of hostilities in Ukraine based on open-source data. It assesses disruption to critical infrastructure, facilities and settlements. The tool is used to track potential problems such as environmental risks and forest fires; it also prioritizes future analysis.

More information can be found here. Impact of war on natural

environment of the Carpathians in Ukraine

21 October 2022

Ministry of Climate and Environment of Poland (Department of Nature

Conservation)

This report provides a brief overview of the protection of the Ukrainian environment, with a particular focus on the Carpathian region. It assesses damages and losses caused by the war, and possible measures to address the most urgent consequences.

More information can be found here. Ecoaction

Live Center for Environmental

Initiatives Ecoaction (Ukraine)

This website is managed by a civil society organization composed of experts and activists. It monitors cases and produces reports relevant to environmental impacts caused by the Russian invasion. It is a partial assessment that should be used to assist Ukrainian authorities after the end of active hostilities. The website includes interactive maps, assessments and reports to further illustrate damage.

More information can be found here. Green post-war

reconstruction of Ukraine: visions and

models August 2022

Resource and Analysis Center (Ukrainian NGO)

This publication is a policy brief offering two green reconstruction models for a sustainable post- war development of Ukraine. The “ambitious” model includes green economy, green growth and environmental preservation. The “pragmatic” model is a green instrument used to mainstream environmental concerns in the decision-making process to prevent long-term negative consequences. (English summary available, full version in Ukrainian).

More information can be found here.

13.06.2023

3

Title Organizations involved Content Conflict and Environment

Observatory: Ukraine Live

Conflict and Environment Observatory (UK) &

Zoï Environment Network (Switzerland)

This website has published reports and briefings pertaining to environmental impacts of the war in Ukraine. Reports include The Coastal and Marine Environment (February 2023), Fossil Fuel Infrastructure (November 2022), Explosive Weapons Use and the Environmental Consequences: Mapping environmental incidents in Ukraine (October 2022), Industry (October 2022), Water (September 2022), Nuclear and Radiation Risks (July 2022), and information collected on possible sustainable recovery data.

More information can be found here. Project: Assessment of

the Environmental Impacts of War against

Ukraine and Options for Remediation

Active since Nov. 2022

OSCE & Zoi Environment Network

(Switzerland) & CEOBS – Conflict and

Environment Observatory (UK)

The project aims to assess the environmental impacts of the war and contribute to the remediation of the damages and the degradation of the environment in Ukraine, in order to address and reduce safety and security risks. The project will assess the overall environmental impacts of war and look at specific sectoral/thematic areas.

Weekly Reports on Environmental Damage

Caused by the War in Ukraine

Since March 2022

Ministry of Environmental Protection and Natural Resources of Ukraine

The Ministry reports weekly on the environmental damage caused by the war in Ukraine. Briefings include, but are not limited to, updates on damage to infrastructure, ecosystems, pollution, freshwater resources, natural reserves and nuclear and radiation safety threats.

More information can be found here.

Risks and impacts from attacks on energy

infrastructure in Ukraine December 2022

NGO PAX (Netherlands) This report provides information on PAX’s monitoring and documenting of incidents from deliberate Russian attacks on energy facilities of Ukraine. As well as analysis of risks and potential impacts posed by these strikes on the environment and public health. The report includes the map of verified damage to energy infrastructure and the case studies, including on Zaporizhzhia Nuclear Power Plant.

More information can be found here. Attacks on Agro- Industrial Sites in

Ukraine April 2023

NGO PAX (Netherlands) This report provides information on PAX’s monitoring and documentation of damage to Ukraine’s agro-industrial facilities. The report outlines the background of Ukraine’s agro-industry and the public health and environmental risks associated with damaged facilities, as well as providing an analysis of the general findings, including a number of case studies focusing on the risks and impacts on the environment in Severodonetsk, Bakhmut and Mayaki.

More information can be found here. Climate Damage Caused

by Russia’s War in Ukraine November 2022

Initiative on GHG accounting of war

This report is an interim assessment focused on climate damage in four activity areas caused by the war in Ukraine: Refugees and IDPs, Warfare, Fires, and Civilian Infrastructure. It focuses on sources of emissions that can be directly linked to the war through these four areas and presents estimated figures for emissions from reconstruction.

More information can be found here.

13.06.2023

4

Title Organizations involved Content First, Second, Third,

Fourth, Fifth and Sixth Ukrainian Energy Sector

Damage Assessments January 2023

(and August, September, October, November and

December 2022)

Energy Charter The damage assessment reports were developed by the Task Force comprised of representatives of Ukrainian authorities and the Energy Charter Secretariat, established under the project "Cooperation for Restoring the Ukrainian Energy Infrastructure" and in cooperation with other Ukrainian and international organizations. The general objective of the project is to assist the Government of Ukraine in the cost -effective restoration of energy infrastructure, considering the clean energy transition while ensuring energy security. The project is funded by the European Commission and implemented by the Energy Charter Secretariat.

More information on the 1st assessment can be found here, the 2nd one here, the 3rd one here, the 4th one here, the 5th one here and the 6th one here.

SaveEcoBot Live

SaveDnipro This website provides Ukrainian citizens the opportunity to submit environmental damage reports and pictures to a chat bot to be collected by the Ministry of Environmental Protection and Natural Resources. SaveEcoBot contains charts of environmental damage, reports on air quality, fire statistics, background radiation, pollution data, and information on inspections. The tool collects information on the sectors of water resources, military waste, industrial accidents, radiation objects, soil, and damage to greenery.

More information can be found here. Environmental

Assessment and Recovery Priorities for

Eastern Ukraine September 2017

OSCE This assessment, conducted before 2017, evaluates the environmental situation in eastern Ukraine. Using the framework of the OSCE Project Coordinator in Ukraine, it extracts analysis from available information to provide practical recommendations for further assessment. The report covers pollution, mine disruptions, water challenges, impact on ecosystems and more. It concludes with short-term and long-term views of the environmental future for eastern Ukraine.

More information can be found here. Impact of the Russia-

Ukraine armed conflict on water resources and

water infrastructure March 2023

Nature Sustainability This analysis provides insight into damage to freshwater resources and water infrastructure caused by the armed conflict during the first three months of the conflict. It identifies the pressures imposed on the water sector; including dams at reservoirs, water supply and treatment systems and subsurface mines. This article also discusses the implications for sustainability in Ukraine and at the global level.

More information can be found here.

13.06.2023

5

Title Organizations involved Content Report on Damages to

Infrastructure Caused by Russia’s War against

Ukraine One Year after the Start of the Full-Scale

Invasion March 2023

Kyiv School of Economics & Ministry of Community and Territorial Development &

Ministry of Infrastructure & Ministry of Health of &

Ministry for Reintegration of the Temporarily

Occupied Territories of & National Bank (all of

Ukraine)

This report presents the results of an assessment of the damage to Ukraine's economy incurred through damage and destruction of physical assets caused by Russia's invasion. The sectors covered are the social sector, the productive sector, infrastructure, the financial sector, the environment and demining. The report covers the period from 24 February 2022 to 24 February 2023.

More information can be found here.

Russian War Against Ukraine: Energy

Dimension Live

DiXi Group DiXi Group is a think tank founded in 2008 in Kyiv. Its purpose consists of research and consultations related to information policy, energy, security and investments. It publishes weekly reviews of the energy dimension of Russia’s invasion of Ukraine, also covering damage to infrastructure.

More information can be found here. Ukraine War

Environmental Consequences Work

Group (UWEC) Live

Ukraine War Environmental

Consequences Work Group (UWEC)

The founders of the Ukraine War Consequences (UWEC) Work Group are environmental activists, experts and journalists. Their goal is to collect, verify, analyse and share information about the invasion, produce expert analyses, and offer development solutions meeting the best environmental standards to address the global humanitarian and environmental crises. The topics covered include transboundary impacts of the invasion, direct environmental impacts, threats and damages to ecosystems and protected areas and species.

More information can be found here. Ukrainian energy sector evaluation and damage

assessment – VIII March 2023

(updated on a monthly basis)

Task Force "Cooperation for Restoring

the Ukrainian Energy Infrastructure" of the

Energy Charter Secretariat

The first Ukrainian energy sector evaluation and damage assessment report was published on 24 August 2022. Since then, the Task Force (comprised of representatives of Ukrainian authorities and the Energy Charter Secretariat) updates monthly the report with information on key energy sector damage. The objective of the project is to assist the Government of Ukraine in the cost-effective restoration of energy infrastructure, taking into account the clean energy transition while ensuring energy security.

More information can be found here. Environmental damage in Ukraine during the full-scale war, 2022

(map)

Greenpeace & Ecoaction

The map presents 30 cases to showcase the most serious damage inflicted in 2022. The cases are categorized by the type of damage and include a short description and satellite imagery of each.

More information can be found here.

13.06.2023

6

Title Organizations involved Content Eastern Ukraine Ecomap

Live Transition Promotion

Program of the Ministry of Foreign Affairs of the Czech

Republic

Map of environmental damage in Eastern Ukraine (in Ukrainian). More information can be found here.

Map of Recovery Live

NGO Anti-Corruption Headquarters (Ukraine)

The Map of Recovery contains information about all civilian infrastructure objects restored, destroyed, damaged partially or completely as a result of the Russian invasion of Ukraine. The Map contains information about each object’s location, approximate period of restoration, date of destruction, restoration estimate, photos before the destruction, the destruction itself, the restoration process and its completion. The source of information about the facts related to the object and the company that has restored it are also reported.

More information can be found here. The Tearline Project

Live National Geospatial-

Intelligence Agency (US) The National Geospatial-Intelligence Agency is partnering with non-profit groups to create public-facing, open-source intelligence on various strategic, economic, and humanitarian intelligence topics that tend to be under-reported within in-depth or long-form formats. It has a section on Ukraine, containing analyses of the environmental impacts of the conflict.

More information can be found here. OSCE Library of publications on

environmental issues in conflict-affected areas

2017-2021

OSCE Collection of publications produced from 2017 until the end of 2021 by the OSCE Project Co- ordinator in Ukraine within its environmental activities in conflict-affected areas in Eastern Ukraine.

More information can be found here.

Map of military installations in Crimea

May 2023

Radio Free Europe/Radio Liberty

The Map includes of more than 200 military installations in Crimea, ground-proofed with satellite imagery. It shows the extent of military land-use and indicates the scale and distribution of the associated environmental impacts. (Map and descriptions are in Ukrainian)

More information can be found here. Ukraine Analysis Hub

Live ACAPS Dashboard of open-source data on damage and access incidents since April 2022. In particular,

the Civilian Infrastructure Damages Dataset tracks damages to civilian infrastructure during the Ukraine conflict, using publicly available sources from various local, regional, and international organisations and media and social media platforms.

More information can be found here. RebuildUA

Live Kyiv School of Economics &

Urbanyna & Culver Aviation &

GeoTema

The RebuildUA project collects drone and satellite images, as well as photo and video recordings of destroyed infrastructure in Ukraine, and it develops infographics and analytical reports containing information on the number, nature, classification of damage and other details for further damage assessment.

More information can be found here.

13.06.2023

7

To propose the addition of more studies and assessments, please complete this questionnaire.