Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Painless data operation in Node.js

Painless data operation in Node.js

Nikita Galkin

June 09, 2018
Tweet

More Decks by Nikita Galkin

Other Decks in Programming

Transcript

  1. Painless data
    operation in Node.js
    at
    Jun 9, 2018

    View full-size slide

  2. Nikita
    Galkin
    Love and Know:
    ▰ How to make developers and business happy
    ▰ Technical and process debt elimination
    Believe that:
    ▰ Any problem must be solved at the right level
    ▰ Software is easy. People are hard
    ▰ A problem should be highlighted, an idea should
    be "sold", a solution should be demonstrated
    Links:
    Site GitHub Twitter Facebook
    2

    View full-size slide

  3. null undefined

    View full-size slide

  4. router.get('/file', function(req, res, next) {
    models.users.findById(req.params.id)
    .then(function(usr) {
    if (!usr) {
    return res.status(404).json({
    msg: 'User not found'
    });
    }
    return res.json(usr);
    })
    .catch(next);
    });

    View full-size slide

  5. Big Response
    time

    View full-size slide

  6. 9
    BE usability factors:
    ▰ response time
    ▰ response time
    ▰ and again
    response time

    View full-size slide

  7. 10
    What affects the response time:
    ▰ network latency
    ▰ Node.js event loop delay
    ▰ Application load (so be ready to
    scale)
    ▰ Bad data flow

    View full-size slide

  8. 11
    Block
    Event loop

    View full-size slide

  9. 12
    How block Event Loop (don’t do it):
    ▰ Use sync operations instead of async
    ▰ Make too many CPU usage
    manipulation in same process
    ▰ Use something like:
    while (Date.now() < end)

    View full-size slide

  10. let last = process.hrtime();
    const CHECK_DELAY_EVERY_SECOND = 1;
    setInterval(() => {
    const time = process.hrtime(last);
    let delay = (time[0] * 1e3 + time[1] / 1e6);
    delay = delay / CHECK_DELAY_EVERY_SECOND - 1000;
    console.log(delay.toFixed(2));
    last = process.hrtime();
    }, CHECK_DELAY_EVERY_SECOND * 1000);

    View full-size slide

  11. 15
    How improve data flow:
    ▰ Use streams
    ▰ Use Promise.all() in async functions
    ▰ Make calculation in DB layer
    ▰ Use precalculated data

    View full-size slide

  12. const balancesP = getBalances(userAddr, tokenAddresses);
    const metricsP = getTokenMetricsObject();
    const pricesP = coinmarketcap.get();
    const [balances, metrics, prices] = await
    Promise.all([balancesP, metricsP, pricesP]);
    // 1-3 seconds
    const balances = await getBalances(userAddr,
    tokenAddresses);
    const metrics = await getTokenMetricsObject();
    const prices = await coinmarketcap.get();
    // 3-9 seconds

    View full-size slide

  13. router.get('/someFile',
    function(req, res, next) {
    fs.createReadStream("./toSomeFile").pipe(res);
    });
    router.get('/users,
    function(req, res, next) {
    users.find({}).pipe(res);
    });

    View full-size slide

  14. CREATE MATERIALIZED VIEW "token_metrics" AS
    SELECT ...
    CREATE OR REPLACE FUNCTION refresh_metrics ()
    RETURNS TRIGGER
    LANGUAGE plpgsql
    AS $$
    BEGIN
    REFRESH MATERIALIZED VIEW token_metrics;
    RETURN NULL;
    END
    $$;
    CREATE TRIGGER trades_updated
    AFTER INSERT OR UPDATE OR DELETE ON trades
    EXECUTE PROCEDURE refresh_metrics();

    View full-size slide

  15. Event Loop
    is Okey

    View full-size slide

  16. DataBase is
    overloaded

    View full-size slide

  17. 21
    How to fix DataBase performance:
    ▰ Use explain and logs
    ▰ Add indexes
    ▰ Add materialized views
    ▰ Add extra data storage as cache
    ▰ Scale!

    View full-size slide

  18. Entity
    vs
    Object

    View full-size slide

  19. 24
    Plain
    Old
    Java
    Object

    View full-size slide

  20. class User {
    constructor(name, surname) {
    this.name = name;
    this.surname = surname;
    }
    }
    const hero = new User('Bart', 'Simpson');
    hero instanceof User; // true
    JSON.stringify(hero); // {"name":"Bart","surname":"Simpson"}

    View full-size slide

  21. // class (constructor) objects
    new User('Bart', 'Simpson');

    View full-size slide

  22. class User {
    constructor(name, surname) {
    this.name = name;
    this.surname = surname;
    }
    }
    const hero = {
    name: 'Gomer',
    surname: 'Simpson'
    };
    hero instanceof User; // false

    View full-size slide

  23. // plain (literal) object
    const hero = {
    name: 'Gomer',
    surname: 'Simpson'
    };

    View full-size slide

  24. 29
    Don't use classes
    instances and plain
    Objects together

    View full-size slide

  25. 30
    1. Use factories
    2. Use class-transformer
    3. Carefully use ORM,
    for example:
    sequelizejs
    typeorm

    View full-size slide

  26. Active Record
    VS
    Data Mapper

    View full-size slide

  27. const User = sequelize.define('user', {
    username: Sequelize.STRING,
    surname: Sequelize.STRING
    });
    // Active record
    User.findAll({where: {age: {Sequelize.Op.gte: 21 } });
    // Repository
    UserRepository.getAllAdult();

    View full-size slide

  28. class User {
    constructor(name, surname, hidden) {
    this.name = name;
    this.surname = surname;
    this.hidden = hidden;
    }
    toJSON() {
    return {
    name: this.name,
    surname: this.surname
    };
    }
    }
    const hero = new User('Bart', 'Simpson', 'iLoveDaddy');
    JSON.stringify(hero); // {"name":"Bart","surname":"Simpson"}

    View full-size slide

  29. class User {
    constructor(name, surname, hidden) {
    this.name = name;
    this.surname = surname;
    this.hidden = hidden;
    }
    toWS() {
    return {
    name: this.name
    };
    }
    }
    const hero = new User('Bart', 'Simpson');
    JSON.stringify(hero.toWS()); // {"name":"Bart"}

    View full-size slide

  30. class User {
    constructor(name, surname, hidden) {
    this.name = name;
    this.surname = surname;
    this.hidden = hidden;
    }
    }
    const hero = new User('Bart', 'Simpson', 'iLoveDaddy');
    JSON.stringify(hero, ['name', 'surname']); // can be function

    View full-size slide

  31. import { Field, ObjectType } from 'type-graphql';
    import { Column, Entity, ObjectIdColumn } from 'typeorm';
    @Entity({ name: 'verbs' })
    @ObjectType({ description: 'Object representing Verb' })
    export class Verb {
    @ObjectIdColumn()
    @Field()
    id: string;
    @Column()
    @Field(type => String)
    name: string;
    }

    View full-size slide

  32. Test Driven Development
    40
    FAIL
    PASS
    TDD
    REFACTORING
    “Repetition of a very short
    development cycle: requirements are
    turned into very specific test cases,
    then the software is improved to pass
    the new tests, only.”
    Kent Beck

    View full-size slide

  33. Types or Documentation Driven Development
    41
    tsc
    Add JSDoc
    TDD
    ▰ choose typescript or JSDoc
    ▰ autocomplete for engineers
    ▰ always actual code documentation
    ▰ puppeteer source-code is
    awesome JSDoc example
    REFACTORING

    View full-size slide

  34. /**
    * @typedef {Object} CoverageEntry
    * @property {string} url
    * @property {string} text
    * @property {!Arrayend: number}>} ranges
    */

    View full-size slide

  35. Theft Driven Development
    43
    STEAL
    ADOPT
    TDD
    SHARE

    View full-size slide

  36. I hope you stole some
    useful ideas from my talk

    View full-size slide

  37. 45
    Thank you for attention!
    Be consistent in your node.js
    data!!!
    You can find me on Twitter as @galk_in
    Slides are available at speakerdeck.com/galkin
    or on my site galk.in

    View full-size slide