Shipping a bug-free feature is always important in every release. To ensure this, we do quality analysis(QA) at various points of the feature cycle. To facilitate an efficient QA, we also maintain certain environments for our app, each serving a different purpose. We have the following environments to be specific:

  1. Production - The actual live app.
  2. Staging - A replica of the production where final sign-off QA is done just before going live.
  3. Test - A quick deployable environment which can be used by developers to share the WIP feature branch with anyone in the company or among other developers.

With multiple features in development simultaneously and multiple environments to deploy, automated deployment becomes very important to ensure frictionless and fast feature lifecycle. In this post, I’ll try to explain how to manage all these environment deployments through automation, especially for our product VWO.

Tests

As mentioned above, tests are very lightweight environments which developers generally create to share their WIP feature branch with other developers, QA or someone from marketing/product to gather feedback. Our app consists of various components: frontend, main-backend and various other micro-services. So each test environment is a combination of different branches from each of the constituent components. For example our app have following components: frontend, backend and Service-1. So our tests can look like:

Test #1 - master (frontend) + feature-notifications (backend) + master (service-1)

Test #2 - feature-auth (frontend) + feature-auth (backend) + master (service-1)

And as these tests should have a unique sharable URL, they can be given names like: feat1.vwo.com or heatmap-optimizations.vwo.com

Deployment

To deploy such a test we have a job on Jenkins. As you may have guessed already, the inputs to this job are:

  1. Name of the test instance
  2. Frontend branch
  3. Backend branch
  4. Service-1 branch

Once this job runs, it pulls on all the above 3 branches on a remote server, does some configuration changes and creates a virtual host to work on testname.vwo.com.

More automation

Now, even this job would require the developer to open Jenkins webapp, go to job page, put in inputs and then run it. But we avoid that too - enter Ramukaka!. Ramukaka is our Skype bot (that we have open-sourced as well) which we use for various grunt tasks, such as running a Jenkins job!

With Ramukaka in the picture, our test deployment looks like so:

Note: We have 3 components and have only 2 branches are specified. That is because the developer can skip a component if the branch to be deployed is default i.e. master. Also, the same command just pulls the latest changes in case the test instance already exists.

Neat, right?

Staging

Staging has primarily 2 differences from test:

  1. There is a single staging unlike multiple tests.
  2. There are some more build steps involved compared to a test.

So it’s similar to a test deployment, except that before deploying it required the developer to build his/her branch like so:

Note: While building a branch we also inform the job about the environment to build for (eg. stagingapp above) because right now the code needs to be a bit tweaked according to the domain its deployed on.

And once Ramukaka confirms a successful build, the developer can deploy the staging with that branch:

Some more commands

As I had mentioned, we have just one staging (single gateway to production). Therefore, each deployment overwrites the previous deployment. And so it becomes important that developers do not overwrite each other’s deployment by mistake. To prevent this, we have an additional command in Ramukaka called currentBranch. Through this command anyone can check which branch is deployed for a particular component on the staging. Eg. if I need to check the frontend branch on staging, I would do so:

Now the developer can take appropriate actions based on the deployed branch.

Production

The production is no different from the staging. Once the final round of testing is done by the QA team on staging, there are 3 things that need to be done to deploy the app on production:

  1. Build the branch
  2. Create a tag for release on master branch
  3. Deploy the tag on the server

All the 3 tasks are handled through a single command on Ramukaka:

And the frontend gets deployed on production, just like that!

Note: Right now only the frontend deployment is automated for production. But we plan to do it for all the components of the app.

Going Ahead

All this deployment automation saves us a huge amount of time. And we know we can save more. Using similar automation for every component of the app is something we plan to do next. Also better logging and monitoring of these environments is on the list.

How do you manage multiple environments? We would love to hear about your deployment techniques if you want to share in the comments.

Until next time!


About PyData

I recently got an opportunity to speak at the PyData, Delhi. PyData is a tech group, with chapters in New Delhi and other regions, where Python enthusiasts share their ideas and projects related to Data Analysis and Machine Learning.

Talks at PyData

There were three talks at PyData, namely Machine Learning using Tensor Flow, Data Layer at Wingify and mine, Learning Data Analysis by Scraping Websites. All the talks were thorough and excellent! In the talk, Data layer at Wingify by Manish Gill 🤓, he talked about how we handle millions of requests at Wingify.

Some of Images of the PyData Meetup Hosted by Wingify.

Background About My Talk

Let me give you a little background. It was the Friday before the PyData Meetup/Conference. Our engineering team was doing its daily tasks. I had just grabbed a coffee to alleviate my laziness. Suddenly, our engineering lead came and asked us whether anyone could present on a topic at the PyData that we were to organise the very next day. An initial speaker, who had confirmed earlier, backed out at the last moment because he had fallen sick. I could see that most of the team members tried to avoid volunteering in such a short notice and also probably because the next day was a Saturday (though this is my personal opinion). But I had something different on my mind and during this planning or confusion, I volunteered for it 🤓. I had a project that I had done, back when I was learning Python. So I offered to present it. He agreed to it and asked me to keep the presentation ready.

Preparing the Project & Slides

That Friday night, I started searching for the old files which I had used. Finally, I found all of them on my website, downloaded them and ran the code. It worked like a charm 😍. Yeah! I quickly created the slides around it, and after finishing, smiled and went to sleep at 4.30 am.

Little About the Basics of My Talk.

The presentation that I gave was on Learning Data Analysis by Scraping Websites. During my college days, we heavily used the BeautifulSoup Library in Python to scrape websites for the many personal projects. During this project, I got the idea to scrape data from the websites which aggregated movies related data. By doing that, I thought that I could create a list of all movies that I must definitely watch. The movies had to satisfy the following criteria:

  1. Release date >= 2000
  2. Rating > 8

It was not the best idea at that time to scrape websites and then analyse(Data frame). But I learned a lot of things by scraping data from the website using Beautifulsoup, then analyzing data using Pandas, visualizing data using MatplotLib (a Python library) and finally coming to conclusion about my movies recommendation.

Coming back to the objective - Finding and sorting the movies released between 2000-2017 in the order of relevance (I didn’t want to watch movies < 2000). Below is the code to scrape IMDB for movies data from 2000-2017.

from bs4 import BeautifulSoup
import urllib2
def main():
    print("** ======  Data Extracting Lib -- by Promode  ===== **")
    testUrl = "http://www.imdb.com/search/title?at=0&count=100&\
    groups=top_1000&release_date=2000,2017&sort=moviemeter"
    pageSource = urllib2.urlopen(testUrl).read()
    soupPKG = BeautifulSoup(pageSource, 'lxml')
    titles = soupPKG.findAll("div",class_='lister-item mode-advanced')
    mymovieslist = []
    mymovies = {}
    for t in titles:
        mymovies = {}
        mymovies['name'] = t.findAll("a")[1].text
        mymovies['year'] = str(t.find("span", "lister-item-year").text).replace('','')
        mymovies['rating'] = float(str(t.find("span", "rating-rating").text)\
        .replace('','')[0:-3])
        mymovies['runtime'] = t.find("span", "runtime").text
        mymovieslist.append(mymovies)
    print mymovieslist
if __name__=="__main__":
    main()

Click here to have a look at the full source code.

You can see the trends like Maximum Rating - Sorted by Rating, Year Vs Rating Trend

DataFrame - Rating is Set as Index


Maximum Rating - Sorted by Rating


Year Vs Rating Trend

Take away from the Talk

With this method, you would have winner’s data from the data set. For example, suppose you want to create a Cricket Team(IPLT20) which has the maximum probability to win the match, what you can do is parse the IPLT20) website for last 5 years’ data and select the top 5 batsmen and 6 bowlers 😎.

Conclusion

I totally understand that this may not be the best project for the data analysis. I am still learning and I showed what I had done. I believe that it served my purpose.

I will be doing more research on data analysis in Python. Thanks for reading this. Below is my talk slides:

Slides:

Slides :-


Introduction

This article will deal with the issues we face with the current API architecture (mostly REST) and why demand-driven APIs seem a perfect replacement for it. We will also talk in brief, about GraphQL and how it is a feasible solution for implementing demand-driven applications.

Note: This article is inspired from Demand driven Applications with GraphQL by Vinci Rufus at JS Channel 2017.

Why Demand-driven API? What’s wrong with REST?

Let’s take a simple example of author & articles. If we are given a requirement to develop an API to fetch authors or articles, it will most probably go like this, if we follow REST:

  • GET /authors/:authorId
  • GET /articles/:articleId

Let’s taken an example where we have to show an article snippet on my website’s dashboard. We would need its title, description & author name. So we hit the latter end point and it will give a response like:

{
  title: 'Demand Driven APIs Using GraphQL',
  createdAt: '2017-04-25',
  updatedAt: '2017-08-25',
  articleId: '96',
  authorId: 50,
  status: 'published',
  description: 'Lorem Ipsum...'
}

There are two problems with this response:

1) Extra information: We only needed the title & description but we got everything related to the article and we cannot get rid of this extra payload as this extra information might be getting consumed at some other page i.e. Edit Article Page.

2) Missing information: We were expecting author name but instead we got authorId. This is bad and to solve this we would probably be making another network call on the former end point to get the author name. It’s an overhead making 2 network calls just to fetch 3 parameters, don’t you think? Also, it will just get more complex as we include more resources i.e. comments, images etc.

How Demand-driven Applications Work?

Now that we understand few issues with REST based APIs, we need a smart system which can give me the exact information required instead of giving me partial/extra information.This can be solved if the client demands what it actually needs and server gives it only that piece of information. This can be done using GraphQL.

Let’s try to solve our problem using GraphQL. The exact information that our client need can be represented in GraphQL as:

{
  article (id: articleId)
  {
    title,
    description,
    author {
      name
    }
  }
}

The server can have a single end point with the following schema:

type Article(id: Integer) {
  title: String,
  description: String,
  status: String,
  createdAt: Date,
  updatedAt: Date,
  status: String,
  author: Author
}

type Author(id: Integer) {
  name: String,
  email: String,
  photo: Picture,
  followers: [User]
}

type Picture(id: Integer) {
  imgPath: String,
  imgHeight: Integer,
  imgWidth: Integer
}

And each field in our schema can have a function to fetch that piece of information. In our case:

  function Article(id) {
    return Article.find(id);
  }

  function Article_title(article) {
    return article.title;
  }

  function Article_description(article) {
    return article.description;
  }

  function Article_author(article) {
    return article.author;
  }

  function Author_name(author) {
    return author.name;
  }

On querying the data, we get i.e.

curl -XGET http://myapp/articles -d "query={
  article(id: 1) {
    title,
    description,
    author {
      name
    }
  }
}"

We will get like this:

{
  title: 'Demand Driven APIs Using GraphQL',
  description: 'Lorem Ipsum...',
  author: {
    name: 'Sahil Batla'
  }
}

This is what we needed, now we can keep the endpoint same and tweak with fields required to display relevant information at any page of our website.

Advantages of Demand-driven APIs

1) Single end point for serving any piece of information.

2) Less payload of data as no extra information is served.

3) Versioning of APIs become simpler as we can control the exact information required.

Disadvantages of Demand-driven APIs

1) Latency may increase due to a single end point handling all the querying of data.

2) No lazy loading possible as it’s a single call which will contain all the data.

Try it Out

If you think GraphQL is promising go ahead and try it out. There is much more to it that you will love to learn. Check out its official documentation. It has been implemented in all the well known languages and you can find it all here.


Have you ever seen a bunch of geeks lock themselves up in a room, hacking throughout the day? This was witnessed when Wingify had its very first Capture The Flag battle.

Capture the Flag (CTF) is a special kind of information security competition which provides a safe and legal way to try your hand at hacking challenges. We have learned a lot of computer science and security concepts in classes, and by reading articles. But participating in a CTF actually teaches how to break into things when they are not implemented properly, which happens all the time in the real world. In this, all you need to do is to find a flag which is a proof that you solved the puzzle, and submitting it to the platform earns your team points. Flags are typically chosen to look very distinctive, so that when you see one, you’ll know that it’s a flag, and that you’ve solved the puzzle. For example, flag{congr4tz_y0u_found_1t}.

Preparation

Sometime back, Facebook open-sourced a platform to host Jeopardy styled CTF competitions and we couldn’t resist ourselves from using it. It’s simply amazing and sleek. It took around 2-3 weeks to prepare for the event and we had fun brainstorming creating the problem set. Creating the problems required thinking of some real world scenarios from the field of software development and security and combine them with references like Mr. Robot, Snowden, etc. A few ideas were taken from prior experience participating in online CTFs and Wargames.

Event

Wingify CTF was an internal event and very first of its type. Bonus points were offered for teaming up with someone from a non-engineering role. We saw some great participation from the customer support, customer success & marketing teams as well. To bring everyone on the same page, participants were asked to register for the event by solving a teaser. And the teaser was to find a flag in a registration form. You’d be surprised to hear that the form was made using Google Forms 😮.

It was an 8-hour long online event which had 45 participants among 16 teams. There was a total of 12 challenges ranging between 40 and 400 based on the difficulty level with total available 1840 points. The set of challenges included problems in web application security and forensics. There was another teaser to be solved before starting off the real game. Early in the CTF, everyone was doing pretty well especially team Matrix and Hunters. In half of the time, quite a good number of hackers were already done with all the problems except the two most difficult ones. When the team Rootcon and Hustlers solved the challenge worth 400 points, they were the clear winners on everyone’s mind. But as they say, it’s not over till it’s over. At the last-minute when team RSS captured that big flag and stood the first place, it was the same feeling like a dramatic last-minute goal in Football.👏

Challenges

I’d like to mention some of the interesting challenges.

  1. XSS - When we talk about Frontend security, cross-site scripting is the first vulnerability that comes to everyone’s mind. One of the challenges was to detect an XSS vulnerability and exploit it by stealing the cookies. The key challenge while creating this problem was using PhantomJS, a headless WebKit, to check whether the XSS payload got successfully triggered. shell_exec('phantomjs fake-browser.js --url' . $url . ' --password ' . getenv('FLAG'));

  2. S3 Secrets/Credentials - This problem was based on the fact that the credentials, such as Amazon S3 keys, Github tokens, and passwords, are often included in published GitHub repositories. Once you have put sensitive data in a Git repository, it is going to stay in the repo’s history forever (there are ways to avoid this).

  3. Encryption - One of my personal favorites was the problem requiring teams to calculate the MD5 of a given string. Sounds pretty straight, right? The challenge is right here in front of you. Can you capture the flag and send it to ctf@wingify.com? 😊

Winners

  1. Team RSS - Rachit Gulati, Sahil Batla, and Sandeep Singh

  2. Team ROOTCON - Gaurav Nanda, Aakansh Gulati, and Ankita Gupta

  3. Team HUSTLERS - Rahul Kumar, Arun Sori, and Dinkar Pundir

Each participant from the top two teams was given Yubikey and Bluetooth Speaker respectively.

Chhavi and I were able to pull off the event successfully. It turned out to be great and everyone had fun hacking together. I would highly recommend doing something like this for your organization. This will surely increase the breadth of security knowledge.

Mini CTF (External)

Last week, Wingify hosted a PyData Meetup and attendees played a quick round of CTF. You can find the pictures below.

If you would like to practice for such events, you should definitely participate in the online CTFs. You can find the list of long-running CTFs. And if you like playing CTFs, we are hiring for Security Engineer position 😍 🙂.


I am a frontend developer at Wingify and I am building a really awesome product, PushCrew. Last month, we had a hackathon. The idea was to ‘Solve Daily Problems’, interesting right? 😃

I am an avid reader and I read a lot of stuff on the web, but I often find myself copying parts of different articles and pasting in my notepad. I always thought that it would be a great idea to have all my summaries at single place. I wanted a platform that could show all the highlighted parts of the articles that I have liked without me having to juggle between different tabs. So instead of waiting for an app like this to be built, I went ahead and created a micro bookmarker at the hackathon.

My idea was simple, and I knew that I could build it alone. So I was a one-person team (Obviously me! 😛 ).

The idea was not just to build, but also to learn something new because that’s the whole purpose of attending a hackathon, right? Since I had never built a Chrome extension before, I started reading about how to build an extension and took some guidance from our in-house frontend God, a.k.a chinchang 😛 . I devoted some good chunk of time to decide my strategy for building the product.

So, after spending an entire night on coke and pizzas, I was able to build a beautiful extension which was working, and solving, at least my problem of highlighting parts of articles that I liked on the web. I really hope it helps a lot of people (read: readers) as well.

Download this awesome application now 🤘 .

Here are a few glimpses of my hack.

Sum It Up demo
Another Sum It Up demo

Prohibited content (Only for geeks):

As soon as the user selects some text on the page (HTML) and right clicks on it, (s)he is shown an option to ‘Save to Sum it up’ in the context menu. On clicking the option, Sum It Up saves the highlighted data (color, text, DOM node, page URL, timestamp etc.) in the JSON format to the local storage (so no breachment of privacy) inside the Chrome browser. The main challenge was to maintain the highlighter for the partial DOM selection which I have solved by putting the custom span tag to all the elements which reside in that selected area.

Some features that you might find useful are:

  1. (High) light it up.
  2. Collect your notes.
  3. Email them.
  4. Searching made easy.
  5. Tweet your note.
  6. Are you a markdown lover? Yes you can export in markdown too.
  7. Directly jump to the micro section of the website.

Sum It Up got featured on Product Hunt too! Yippee :) (My very first submission on Product Hunt and that too got featured, it’s like Diwali Bonus 😀 )

PS: This is my first blog post so please be kind to me. I am open to any feedback 😀