Friday, November 06, 2015

OAI, Swagger and the way to build your API's

There is an evolving and trending tool called swagger, which is a simple yet powerful representation of your RESTful API, With the largest ecosystem of API tooling on the planet!

If you are asking whether this tool is promising, Well linux foundation who took the lead on the open API initiative (OAI), consider swagger 2.0 Spec to be the base for it and is going to extend it, this is a serious decision (see here)

This means that when using that specification to describe your API, you get the advantage of the following for free!!!:
  • Generate your server code
  • Generate your client SDK code (multiple languages of course) - Increase adoption of your service
  • Full blown editor for your API's specifications, lets call this the architects whiteboard (it also has tools embedded in it)
  • Generate online documentation for your API
  • Many services already support automatically integrating with API that expose this specification (e.g. Camel and other)
Cool ha ? (this is a time saver and a serious easy-to-do-business factor)

There are couple of ways you can start 
  1. Top down - You start with describing your API and than generating using the above
  2. Bottom up - You already have your API, so either you use some tools to generate the specification or build it based on your API and than use the tools
Below is an example of little PET API: (no additional explanation needed)

swagger: '2.0'
  version: 1.0.0
  title: Swagger Petstore (Simple)
  description: A sample API that uses a petstore as an example to demonstrate features in the swagger-2.0 specification
  termsOfService: ''
    name: Swagger API team
    url: ''
    name: MIT
    url: ''
basePath: /api
  - http
  - application/json
  - application/json
      description: Returns all pets from the system that the user has access to
      operationId: findPets
        - application/json
        - application/xml
        - text/xml
        - text/html
        - name: tags
          in: query
          description: tags to filter by
          required: false
          type: array
            type: string
          collectionFormat: csv
        - name: limit
          in: query
          description: maximum number of results to return
          required: false
          type: integer
          format: int32
          description: pet response
            type: array
              $ref: '#/definitions/pet'
          description: unexpected error
            $ref: '#/definitions/errorModel'
      description: 'Returns a user based on a single ID, if the user does not have access to the pet'
      operationId: findPetById
        - application/json
        - application/xml
        - text/xml
        - text/html
        - name: id
          in: path
          description: ID of pet to fetch
          required: true
          type: integer
          format: int64
          description: pet response
            $ref: '#/definitions/pet'
    type: object
      - id
      - name
        type: integer
        format: int64
        type: string
        type: string

Wednesday, November 04, 2015

Mac OS El Capitan, Python and SIP - arrrrggggg

Got my new mac and a simple task for installing GEVENT / other python modules is breaking with access denied (even when using SUDO)....

This is occurring since python was provided by Apple and its folders are protected by the System Integrity protection, and usually people try to install packages to the global (rather than local)

The best solution I found till now is built out of two steps

  1. PyEnv - ability to install different version of python in your system
  2. PyEnv-virtualenvs - ability to have separate packages folders (isolate)
Installation is a breeze using home-brew and please don't forget that you have 2 manual step (to update the .profile)

<Win-vs-Mac /> - Tell you the truth, today experience reminded me of the struggle we all had when we moved to windows 7/windows 2008 with the UAC.... 

Side note - it's true that you can install to local user (using --user arg) but I found above solution to be much better,

Wednesday, October 28, 2015

gevent - a concurrency framework for python

I am sure that part of your code is filled with IO related tasks which majority of the time consume most of the execution duration... so scale it.

Before you say, "Easy, I just add more threads and everything is running on parallel and faster", threads has cost of spawn time, memory footprint --> bringing limit to your scale!

a better solution might be to make your application not block on your IO operations (e.g use that "dead time"), and for that you can use several framework like concurrent.future, eventlet, gevent and other

In one of my new utilities (API crawler) that is highly impacted by IO procedure (since all it does, is query and wait), I chose to use GEVENT to make my code run concurrent and better utilize my system resources - the outcome was amazing!!!

At the end of the day its all a matter of what are your requirements, e.g. you might still need to spawn tasks into a distributed queue that handles those in a cluster, but that doesn't mean that each worker cannot consume task and run the, concurrent --> use hybrid solution (Such as celery for python) which maximize your throughput and still take advantage of persistency of tasks, Error handling, etc

p.s. I found a very good site which shows great real life example of usage of gevent here

Check out the google trend:

Gotta love those code comments :)

I found one comment which made me laugh while reviewing Etsy's SkyLine Project

# Keep yourself occupied, sucka
while 1:

For those of you who doesn't know what Sucka means: it mean bitch in an angry way like... shut up BITCH

Monday, October 26, 2015

GData API navigation is limited to TOP 500 results (e.g. you cannot navigate more than 10 times assuming 50 results per page)

The number of results you tube API is providing (potentially also all other GDATA) is an estimation and as such if you got totalResults = 4000 , the API will only allow you to get the first 500 (and not w/a for it).
When iterating on the next page, you will see that on the 10th page (assuming 50 results per page) the nextPageToken will be null.

The main reason is that the API is estimating and also ensuring quality of search for the first segment

So if you need to get more than 500 results, suggest you partition the queries in some way

See here for original post

Thursday, September 17, 2015

Installing Windows 10 on Intel NUC without Bootable Media or any tool

I just bought my kids NUC I5 (the new edition) and wanted to install windows 10 on it, since i don't have external DVD i decided to create a bootable USB

I connected my SATA mini HDD using Apricon SATA Wire USB cable, and sadly no matter what tools I used DIDN'T recognize my external disk as removable media, and as such, didn't allow me to create the bootable USB :( (I even tried the great app called rufus)

At the end what I did was:

  1. Connect the external Disk (shows as local disk)
  2. Format as exFAT
  3. Extract my windows 10 ISO to the main volume on the external disk
  4. Connect the External disc using the Apricon SATA Wire to the NUC
  5. Restart and press F10 (for boot options)
  6. Choose my External disk
All is working without ANY TOOL.

Tuesday, August 11, 2015

Managers Making Mess

The time it takes to code a feature is consist of the following formula MMMf(D(x)+Q(x))

x - feature
D(x) - Time to develop (this include CI/CD in it)
Q(x) - Time to test outside of automation
MMMf(...) - Managers Making Mess factor - this is the amount of noise introduced by additional layer in the organization

What surprise me me every day, is the large factor of noise introduced by other parties outside of the dev unit.

Think where would we be if we only needed to implement it without the extra noise... everything would be so simple !

So what do you do to reduce the MMMf to 1.0 ?

Monday, July 20, 2015

Prevent the installer from displaying passwords or other confidential information in the log file.

Most of us know about the article that talks about how preventing Confidential Information from Being Written into the Log File, but few more questions are raised in that area:

  1. Can we pass the list of properties we want to hide (e.g. MsiHiddenProperties property) using the command line (instead of changing our MSI) - Surprising the answer is NO, this has to be built inside the MSI as new property and compile it
  2. If we follow all the steps identified in the main article, Will the Windows Installer hide all the sensitive information we request from being written to all log lines - NO, for example if you saved the sensitive info to registry, the operation will be in the log (But than again, WHY would yo save clear text)
    Another example is that if your code writes to log than you will have to manage it
as a rule of thumb, you should NEVER persist the password as clear text in any area and I would even question the need to persist it in any case, find alternative like delegation / SSO / SSH keys and other which will be more robust (cost more) and more secure 

Hope this helps