Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Switch ML, Code BEAM Europe 2022

Switch ML, Code BEAM Europe 2022

Smashing the data bottleneck with Federated Machine Learning on the BEAM

Vijay Chakilam

May 19, 2022
Tweet

Other Decks in Programming

Transcript

  1. Agenda • Machine Learning • The Data Bottleneck • Federated

    Machine Learning • Challenges • Phoenix + Pyrlang + EpicBox • Algorithms • Future Work • References
  2. Machine Learning • Extract patterns in the input data •

    Form rules based on the patterns • Apply the rules on new incoming data points • Output predictions
  3. Deep (Machine) Learning • Compose a chain of boxes that

    each learn their own set of rules based on the input data points • The output of one box is the input for the next box in the chain • Capable of constructing higher level patterns from low level patterns by way of composition
  4. Capra, Maurizio & Peloso, Riccardo & Masera, Guido & Ruo

    Roch, Massimo & Martina, Maurizio. (2019). Edge Computing: A Survey On the Hardware Requirements in the Internet of Things World. Future Internet. 11. 100. 10.3390/fi11040100.
  5. • Federated learning is a type of distributed machine learning

    • Federated learning systems rely on a central server to coordinate the model training process on multiple, distributed client devices where the data are stored • The model training is performed locally on the client devices, without moving the data out of the client devices
  6. Motivation Lo, Sin Kit et al. “A Systematic Literature Review

    on Federated Machine Learning.” ACM Computing Surveys (CSUR) 54 (2021): 1 - 39.
  7. Message Channel Handlers def handle_in("create", payload, socket) do %{"script" =>

    script, "strategy" => strategy} = payload push_to_stack(script, strategy) {:noreply, socket} end def handle_in("execute", payload, socket) do %{"input" => input, "strategy" => strategy} = payload response = get_from_stack(input, strategy) broadcast(socket, "execute", response) {:noreply, socket} end
  8. Data structure to store Aggregation strategies and their state defmodule

    Pyrlangpoc.Stack do use Agent def start_link(_) do # __MODULE__ Agent.start_link(fn -> %{} end, name: __MODULE__) end def put(key, value) do Agent.update(__MODULE__, &Map.put(&1, key, value)) end def get(key) do Agent.get(__MODULE__, &Map.get(&1, key)) end end
  9. Phoenix Pyrlang def get_from_stack(input, strategy) do message = Stack.get(strategy) my_process

    = {:my_process, :"[email protected]"} res = GenServer.call( my_process, Jason.encode!(%{input: input, oldvalue: message[:state], script: message[:script]}) ) response = Jason.decode!(res) if Map.get(response, "stdout") != "" do Stack.put(strategy, %{script: message[:script],state: Map.get(response, "stdout")}) end response end
  10. Pyrlang Node class MyProcess(GenServer): def __init__(self, node) -> None: super().__init__()

    node.register_name(self, Atom("my_process")) @call(1, lambda msg: True) def hello_catch_all(self, msg): """Catch all handler""" data = json.loads(msg.decode()) script = data.get("script") input = data.get("input") # convert from string data type to its own data type input = ast.literal_eval(input) oldvalue = data.get("oldvalue", None) main_script = """import argparse import ast parser = argparse.ArgumentParser() parser.add_argument("--input", "-i", help="User Value") args = parser.parse_args() input = ast.literal_eval(args.input) {script} """.format(script=script)
  11. profiles = { "python": { "docker_image": "pyrlangpoc_ml- poc:latest", } }

    # docker composed image epicbox.configure(profiles=profiles) cmd = f"python3 {file_data.get('name')} --input '{input}'" result = epicbox.run( "python", cmd, files=[file_data], limits=limits, ) EpicBox
  12. Random Aggregation McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.;

    Arcas, B.A.Y. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 2017; Singh, A.; Zhu, J., Eds.; Volume 54, PMLR, Fort Lauderdale, FL, USA; pp. 1273–1282​
  13. Selective Aggregation D. Ye, R. Yu, M. Pan, and Z.

    Han. 2020. Federated Learning in Vehicular Edge Computing: A Selective Model Aggregation Approach. IEEE Access (2020), 1–1.​
  14. FedProx Aggregation Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar

    Sanjabi, Ameet Talwalkar, and Virginia Smith. 2018. Federated Optimization inHeterogeneous Networks. arXiv:1812.06127 [cs.LG]​
  15. Future Work • Aggregation Algorithms • Private Machine Learning •

    Differential Privacy • Personalization • Evaluation Metrics
  16. References • https://github.com/switch-ml/phoenix-pyrlang-poc • Lo, Sin Kit et al. “A

    Systematic Literature Review on Federated Machine Learning.” ACM Computing Surveys (CSUR) 54 (2021): 1 - 39. • McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; Arcas, B.A.Y. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 2017; Singh, A.; Zhu, J., Eds.; Volume 54, PMLR, Fort Lauderdale, FL, USA; pp. 1273–1282 • D. Ye, R. Yu, M. Pan, and Z. Han. 2020. Federated Learning in Vehicular Edge Computing: A Selective Model Aggregation Approach. IEEE Access (2020), 1–1. • Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2018. Federated Optimization in Heterogeneous Networks. arXiv:1812.06127 [cs.LG]