IEEE Menu Close
Publications lead hero image abstract pattern

Publications

Smart Grid? Yes, AI Says: Bring It on!

IEEE CTN
Written By:

Melike Erol-Kantarci, University of Ottawa, Canada

Published: 21 Apr 2021

network

CTN Issue: April 2021

A note from the editors:

Remember the days when power grid was just the wires to your home?  Have you had the pleasure of having an energy trader knock on your door or spam call your home?   Welcome to the world of “smart power distributions” or “smart grid”.  Smart grids can trade electricity with the consumers and that has become possible with the advent of advanced metering and sensing technologies, e.g., Industrial IoT, new distributed energy resources and ultra-ultra-reliable low latent 5G wireless communication. These smart grids can reduce or remove the carbon footprint against long-haul energy transmissions from national grids instead they meet the electricity demand locally through intelligent demand-supply matching algorithms or games. To deliver this vision, smart grid systems need ubiquitous, mission critical type connectivity solutions to estimate the energy demand accurately and make decisions in real time.

In this article, Melike provides an overview of the legacy power grid followed by evolution of smart grid and their emerging needs. Later, she presents the role of emerging technologies such as artificial intelligence, machine learning and 5G communications that allow these smart grid and associated technologies to make real time decisions and trade electricity, meeting the demand locally. She argues that efficient physical layer techniques, such as joint resource block allocation and user association are required to ensure reasonable connectivity for microgrid and mobile users in the network.

Enjoy the article and send us your comments.

Muhammad Zeeshan Shakir, Editor

Miguel Dajer, Editor-in-Chief

AI-enabled Transactive Energy Systems and the Role of AI-enabled Communications

Melike Erol-Kantarci

Melike Erol-Kantarci

Transactive Energy Systems and Peer-to-Peer Energy Trading in the Smart Grid

Microgrids and battery technologies have been the key enablers of energy trading in smart grids. Energy trading, or in other words energy transactions among prosumers; and between a prosumer and the electric utility, eventually led to the ideas around transactive energy systems that are under consideration for future smart grids.

Before we talk about microgrids, energy trading, smart grid, and so on, it would be good to take a look back and remember how the legacy power grid was. Legacy power grid, or the electricity grid before smart grid, was unidirectional. Electricity generated at plants would be transported over long haul transmission lines to the distribution system, and from there on, electricity would be distributed to the customer premises. Usage would be monitored by meters which had to be read by crews manually. There was no communication between customers and the utilities, and limited information were polled from the field to monitor the health of power lines and transformers. Smart grid introduced many novelties starting with Advanced Metering Infrastructure (AMI) which meant remote meter readings and opened up the communication channel among customers and utilities. Once two-way communication was set up, this quickly led to the developments around intelligent ways of doing load control, load curtailment and energy management for customers. More accurate data collection with Phasor Measurement Units (PMUs), efficient renewable energy generation and integration, new storage technologies and integration of EVs are just among the other novelties in smart grid that lined up after AMI. However, it is hard to say all these advances are completely mature and commercially deployed in all power grids around the globe today. Due to immense investments required and the infeasibility of tearing down the legacy grid and setting up a new grid, most of the advances or the dreams of smart grid are either partially implemented or yet be implemented.

Now, coming back to microgrids, in fact, they date before the concept of smart grid emerged. In 2002, Lasseter explored the deployment of microgrids for the military [1]. Civilian use cases had to wait for the smart grid concept to emerge, however. Today, there are many examples of microgrids in the campus scale, neighbourhood scale, or even as a single building (e.g., IIT Microgrid and the University of Aalborg Microgrid). The fundamental feature of a microgrid, is its capability of islanding. Islanding from the supplier implies there is generation capacity to supply loads within the microgrid. If the generation is renewable, then energy storage is another crucial component of a microgrids. Formerly, microgrids were considered for protection, so if there were faults in the utility grid, a resilient microgrid would sustain using its own supply after islanding from the main grid. Nowadays, most microgrids can disconnect from the grid unless they need to import or export energy from the grid. In some studies, researchers took this self-sustaining idea to a group of microgrids, where by trading energy among themselves, and by forming an overlay network, they could island from the grid as a group or community [2]. It would even be possible to consider them as coalitions [3]. As microgrids might not have the electrical distribution lines among them, they can perform transactions with the grid and hence the power flow will be to and from a microgrid to the utility grid, while the monetary transactions will be among peering microgrids. On a small scale, peer to peer energy trading has been possible to implement in real world cases such as in Brooklyn, NY.  Brooklyn Microgrid defines itself as an energy marketplace for locally-generated, solar energy where prosumers generate electricity with PV, store and sell it to their neighbors using blockchain technology.

In parallel to the advances in smart grid and microgrids, EV technology has been developing fast since 2010s. EV’s charging/discharging capability eventually lead to the idea that their battery could be used to balance out the energy transactions, if a microgrid was tight with supply or had surplus. Hence, EV discharging capability made it possible for EV energy transactions in the transactive energy systems.

Finally, in the modernization path of the grid, it is now almost inevitable to talk about how AI is penetrating into the smart grid technologies. One of the main drivers for AI is access to lots of data. Machine learning algorithms can provide analytics over the collected data from the smart grid, and help to determine the health of the smart grid, provide better forecasting than before and even offer control over loads or services. In this aspect, deep learning has been an emerging star in forecasting loads, price or renewable generation so far. However, this is the tip of the iceberg. Cutting-edge research aims to use reinforcement learning algorithms to introduce control, maybe not in real time, but for non-real-time (time ahead) pricing and energy management.

AI-enabled Transactive Energy Systems

In many prior studies, energy trading has been modeled using various game theory methods. The list is long but just to give some examples we have included some studies here. In [4], a two-level continuous kernel Stackelberg game is proposed for distributed energy trading between microgrids, in which seller and buyer microgrids are classified as leader and follower agents, respectively. In [5], the authors proposed a coalitional game theory scheme to solve the problem of energy management in local energy communities. A comprehensive summary of game theory based techniques can be found in [6].

More recently, similar problems have been tackled from the lens of AI. Machine learning algorithms can adapt and learn the environment. To this end, they have been used to optimize energy trading decisions of microgrids. For instance, in [7] a reinforcement learning based energy trading game has been proposed. In our work  [8], we aimed to minimize power loss within microgrid coalitions while addressing the uncertainties from the energy level of agents in the system using Bayesian learning. In Figure 1, we show how potential coalitions are formed and then how they dynamically change over time.

Figure 1: Dynamic microgrid coalitions.
Figure 1: Dynamic microgrid coalitions.

Even before considering uncertainty, energy management in a group of microgrids is a challenging problem. Several studies have looked into using deep reinforcement learning to solve this problem [9-11]. In our work [12], we considered that a microgrid can be modelled as multi-agent system where storage, generation and load are each represented by an agent. The agents might be controlled by a single aggregator or they may have different owners or controllers. In this setting, a renewable generation unit (RER) can sell energy to the grid or to the storage unit (ESS). The decision of who to sell to and the price of selling will determine the revenue for the renewable agent. In this case, the agents aimed to maximize their own profit, and the competition increased the difficulty of balancing the profit. In addition, we considered deferrable appliances which helped our system to learn the best times to run the appliances. We proposed a correlated deep Q-learning method for the microgrid energy management, where each agent runs the deep Q network independently and then correlated equilibrium is used for coordination. Figure 2 gives an insight on energy trading decisions of the agents. For instance, the ESS agent uses PV power to charge at times 12, 13 and 14, and the stored electricity is sold to demand controlled appliances (DSM devices) at times 20 and 21. The operation time of deferrable devices is presented in Figure 3. Devices in categories 2 and 3 are deferred 4 and 5 hours, respectively, to use the lower price of PV power. Device categories 4 and 5 are deferred 5 and 1 hours, respectively, to buy electricity from the ESS agent. Device 1 is not deferred because the operation limit is in the lowest price period.

Besides control approaches for managing supply, storage and demand, reinforcement learning has been used for decision making in electricity markets.  Peters et al. propose autonomous broker agents that use reinforcement learning, learn market pricing and make profit-maximizing decisions in energy trading [13].

Figure 2: Optimal energy trading under correlated deep Q-learning.
Figure 2: Optimal energy trading under correlated deep Q-learning.
Figure 3: Device operation times under correlated deep Q-learning.
Figure 3: Device operation times under correlated deep Q-learning.

In summary, the nexus of smart grid optimization using AI is just blooming. We will now flip the coin and look at the communication aspect of smart grid and explain how AI-enabled communications is making the grid smarter. 

AI-enabled Communications and 5G for the Smart Grid

Making the power grid smart, mostly refers to connect and compute capability, at least, as an enabler of all other smart things to emerge. Many communication technologies have been considered for the smart grid. In the legacy grid, proprietary technologies were dominant however the abundance of commercial alternatives seems to dominate the landscape nowadays. From Power Line Communications (PLC) to the wireless alternatives such as WiFi, Zigbee and LTE, there is a wide variety of technologies to choose from. Wireless technologies have the ease of integration with other consumer products and they are ubiquitous. In particular, device-to-device (D2D) techniques in LTE has been promising. Based on this, we developed an AI-enabled resource allocation technique that would allow smart grid devices to communicate with low-latency [14]. Most of the times, low-latency is required for near-real-time control applications [15]. To test the performance we also developed a co-simulator of power systems and communication systems [16] and shared it with the community at NETCORE github page. The high-level system diagram shown in Fig. 4 shows the message passing attributes of the developed software.

Figure 4: High-level system diagram of the co-simulator [17].
Figure 4: High-level system diagram of the co-simulator [17].

With the deployments of 5G all around the world and the fact that 5G natively supports Ultra-Reliable Low-Latency Communications (URLLC), applications such as self-driving cars and IIoT, it is anticipated that it would create a potential for unified solutions for the smart grid connectivity. 5G-NR identifies three service categories: enhanced Mobile Broad Band (eMBB), massive Machine-Type Communication (mMTC) and URLLC. Traffic of URLLC users is sparse with short packet size which requires rapid scheduling decisions for achieving close to 1 ms latency. In [15], we proposed a Delay Minimizing Deep Q-Network (DM-DQN) that aims at finding an efficient joint RB allocation and user association in a wireless networks that serves regular UEs as well as microgrids as uRLLC users. There are many other examples of research in the domain of low-latency smart grid communications. The concept of Energy Internet is demonstrating the need for smart grid communications at all timescales, including applications with requirements of several minutes of latency to near millisecond latency [17].   Fueled by all these advances in 5G, and a growing enthusiasm in industrial settings to adopt 5G, we believe utilities won’t be an exception. As AI-enabled communications is starting to shape the direction of 6G, an AI-enabled networking infrastructure for the future AI-enabled power grids doesn’t seem so distant.

References

  1. R. H. Lasseter, "MicroGrids," in Proc. of IEEE Power Engineering Society Winter Meeting, pp. 305-308, 2002.
  2. M. Erol-Kantarci, B. Kantarci, H. T. Mouftah, "Reliable Overlay Topology Design for the Smart Microgrid Network," IEEE Network, Special issue on Communication Infrastructures for Smart Grid, vol. 25, no.5, pp.38-43, September/October 2011.
  3. W. Saad, Z. Han and H. V. Poor, "Coalitional Game Theory for Cooperative Micro-Grid Distribution Networks," in Proc. of IEEE International Conference on Communications Workshops (ICC), Kyoto, 2011.
  4. J. Lee, J. Guo, J.K. Choi, M. Zukerman, “Distributed Energy Trading in Microgrids: A Game-Theoretic Model and Its Equilibrium Analysis,” IEEE Trans. on Industrial Electronics, vol. 62, 3524–3533, 2015.
  5. C. Feng, F. Wen, S. You, Z. Li, F.  Shahnia, M.  Shahidehpour, “Coalitional Game Based Transactive Energy Management in Local Energy Communities,” IEEE Trans. on Power Systems, 2019.
  6. W. Saad, Z. Han, H. V. Poor and T. Basar, "Game-Theoretic Methods for the Smart Grid: An Overview of Microgrid Systems, Demand-Side Management, and Smart Grid Communications," in IEEE Signal Processing Magazine, vol. 29, no. 5, pp. 86-105, Sept. 2012.
  7. X. Lu, X. Xiao, L. Xiao, C. Dai, M. Peng and H. V. Poor, "Reinforcement Learning-Based Microgrid Energy Trading With a Reduced Power Plant Schedule," in IEEE Internet of Things Journal, vol. 6, no. 6, pp. 10728-10737, Dec. 2019.
  8. M. Sadeghi, M. Erol-Kantarci, “Power Loss Minimization in Microgrids Using Bayesian Reinforcement Learning with Coalition Formation,” IEEE 30th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), 2019, pp. 1–6.
  9. T. Chen, and S. Bu, “Realistic Peer-to-Peer Energy Trading Model for Microgrids using Deep Reinforcement Learning,” in Proc. of IEEE PES Innovative Smart Grid Technologies Europe, pp.1-5, Bucharest, Romania, Sep. 2019.
  10. T. Jiang, “Deep Reinforcement Learning for Smart Home Energy Management,” vol.7, no.4, pp. 2751-2762, Dec. 2019.
  11. V. Bui, A. Hussain, and H. Kim, “Double Deep Q-Learning-Based Distributed Operation of Battery Energy Storage System Considering Uncertainties,” IEEE Trans. Smart Grid, vol.11, no.1, pp. 457 – 469, Jan. 2020.
  12. H. Zhou, M. Erol-Kantarci, “'Correlated Deep Q-learning based Microgrid Energy Management,” in Proc. of  IEEE International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD), September 2020.
  13. Peters, M., Ketter, W., Saar-Tsechansky, M. et al., “A reinforcement learning approach to autonomous decision-making in smart electricity markets,” Machine Learning vol. 92, pp. 5–39, 2013.
  14. K. Shimotakahara, M. Elsayed, K. Hinzer, Melike Erol-Kantarci, “High-Reliability Multi-Agent Q-Learning-based Scheduling for LTE-Compliant D2D Microgrid Communications,” IEEE Access, vol. 7, no. 1, pp. 74412-74421, December 2019.
  15. M. Elsayed, M. Erol-Kantarci, B. Kantarci, L. Wu, J. Li, “Low-latency Communications for Community Resilience Microgrids: A Reinforcement Learning Approach,” in IEEE Transactions on Smart Grid, vol. 11, no.2, pp. 1091-1099, 2020.
  16. K. Shimotakahara, M. Elsayed, K. Hinzer, M. Erol-Kantarci, “Mobile Communications-Enabled Smart Grid Co-Simulator System Design," IEEE Systems Journal, June 2020.
  17. H. M. Hussain, A. Narayanan, P. H. J. Nardelli and Y. Yang, "What is Energy Internet? Concepts, Technologies, and Future Directions," in IEEE Access, vol. 8, pp. 183127-183145, 2020.

Statements and opinions given in a work published by the IEEE or the IEEE Communications Society are the expressions of the author(s). Responsibility for the content of published articles rests upon the authors(s), not IEEE nor the IEEE Communications Society.

Sign In to Comment