If you’re interested in Cloud Computing, Amazon Web Services (AWS), Public Cloud, Private Cloud, OpenStack and other technologies and want to meet other people that are as well, check out the group - http://www.awsmichigan.org.

Event: AWS Michigan Meetup

Event Date: Tuesday, December 11th, 2012 @ 7:00pm

More info: http://www.awsmichigan.org/events/94452352/

Author information

Eric Detterman

Founder and CEO at Solid Logic Technology Inc.

Interested in Web Applications, Cloud Infrastructure, Quantitative Trading Systems, and more. Owner of an awesome St. Bernard named Barnaby.

TwitterGoogle+LinkedIn

Cloud Computing Deployment Models

Cloud computing can broadly be broken down into three main categories based on the deployment model. Here is a definition of each one, courtesy of Wikipedia:

Public Cloud

Public cloud applications, storage, and other resources are made available to the general public by a service provider. These services are free or offered on a pay-per-use model. Generally, public cloud service providers like Amazon AWS, Microsoft and Google own and operate the infrastructure and offer access only via Internet (direct connectivity is not offered).^[28]

Private Cloud

Private cloud is cloud infrastructure operated solely for a single organization, whether managed internally or by a third-party and hosted internally or externally.^[4] Undertaking a private cloud project requires a significant level and degree of engagement to virtualize the business environment, and it will require the organization to reevaluate decisions about existing resources. When it is done right, it can have a positive impact on a business, but every one of the steps in the project raises security issues that must be addressed in order to avoid serious vulnerabilities.^[53]

Hybrid Cloud

Hybrid cloud is a composition of two or more clouds (private, community or public) that remain unique entities but are bound together, offering the benefits of multiple deployment models.^[4] By utilizing “hybrid cloud” architecture, companies and individuals are able to obtain degrees of fault tolerance combined with locally immediate usability without dependency on internet connectivity. Hybrid cloud architecture requires both on-premises resources and off-site (remote) server-based cloud infrastructure. Hybrid clouds lack the flexibility, security and certainty of in-house applications.^[52] Hybrid cloud provides the flexibility of in house applications with the fault tolerance and scalability of cloud based services.

Evaluting Private and Hybrid Cloud Solutions

We’re currently involved in several projects that require a hybrid cloud approach. Under this approach, some processing or storage would be handled inside the organization’s data center (private cloud) and other processing or storage would be handled using a public cloud such as Amazon Web Services or Rackspace Cloud. The decision on what workloads to run on the public cloud is generally driven by compliance requirements. An example would be: development and test systems, which use dummy data are candidates for public cloud where the production system is not due to data compliance requirements.

This type of approach is becoming more common and could be viewed as an improvement upon previous IT hardware virtualization initiatives.

In this post we’re going to describe the initial steps to evaluate existing private cloud solutions. This post does not dive into the technical differences between the software solutions, but rather looks at them from the standpoint of – which organization appears to be the most likely to become the market leader.

Private Cloud Computing Software

There are three primary software solutions in the private cloud space – Openstack, Cloudstack, Eucalyptus.  Each have their strengths and weaknesses and are complex products. Relative to other IT industry trends, cloud computing is still relatively immature. Relative to the public cloud, the concept of private cloud is even more immature.

Below is a quick overview of each of the projects.

Openstack

From their website:

About OpenStack

OpenStack is a cloud operating system that controls large pools of compute, storage, and networking resources throughout a datacenter, all managed through a dashboard that gives administrators control while empowering their users to provision resources through a web interface.

Learn more about OpenStack’s compute, storage, networking, take a tour of the dashboard, or dive in and learn how to get started with OpenStack today.

Cloudstack

From their website:

What is CloudStack?

Apache CloudStack (Incubating) is software designed to deploy and manage large networks of virtual machines, as a highly available, highly scalable Infrastructure as a Service (IaaS) cloud computing platform. CloudStack is used by a number of service providers to offer public cloud services, and by many companies to provide an on-premises (private) cloud offering, or as part of a hybrid cloud solution.

CloudStack is a turnkey solution that includes the entire “stack” of features most organizations want with an IaaS cloud: compute orchestration, Network-as-a-Service, user and account management, a full and open native API, resource accounting, and a first-class User Interface (UI).

CloudStack currently supports the most popular hypervisors: VMware, KVM, XenServer and Xen Cloud Platform (XCP).

Users can manage their cloud with an easy to use Web interface, command line tools, and/or a full-featured RESTful API. In addition, CloudStack provides an API that’s compatible with AWS EC2 and S3 for organizations that wish to deploy hybrid clouds.

Eucalyptus

From their website:

What is Eucalyptus?

Openstack vs. Cloudstack vs. Eucalyptus: Quick Analysis

We’re always analyzing and reviewing new and upcoming technologies. One of the best ways to do this quickly is to simply look at Google Trends. This tool gives you a quick way to get a line graph of search trends over time, and drill down by different factors. This method can be done very quickly and works for almost any technology trend that is, or is going, mainstream.

We compared openstack vs cloudstack vs eucalyptus vs amazon aws. We only included searches over the past twelve months from the United States in the “Computers & Electronics” category. Here is a link to the full report and here are the results:

Interest over time:

Regional interest:

Openstack:

Cloudstack:

Eucalyptus:

Amazon aws:

Related terms:

Openstack:

Cloudstack:

Eucalyptus:

Amazon aws:

Conclusions

The main takeways from this is that, all technical capabilities aside, Openstack is far more popular than its two main competitors. This is especially true once you get outside of California. While its far too early to say who will win the private cloud platform war, its pointing towards Openstack. Its good to see some consistency across the US for both Openstack and Amazon Web Services (AWS).

The question that a development company must ask is: Does it matter if there are other solutions that are technically superior, if the target market hasn’t heard of them?

Because of the Google Trends results and the fact that our business centers around Metro Detroit, we’re primarily focusing on Openstack for private cloud and Amazon Web Services (AWS) for public cloud solutions.  We’ll still be working with the other solutions, but will likely be looking at them in a different light going forward.

Related articles

• 2013: Year of the hybrid cloud
• Eucalyptus CEO Mickos: “We Each Have Our Own Sweet Spot”
• Citrix’s CloudStack Cloud Platform Is No Also-Ran [OStatic]
• Eucalyptus’ AWS Bet

Author information

Eric Detterman

Founder and CEO at Solid Logic Technology Inc.

Interested in Web Applications, Cloud Infrastructure, Quantitative Trading Systems, and more. Owner of an awesome St. Bernard named Barnaby.

TwitterGoogle+LinkedIn

Most common cloud computing questions

The most common question we hear about cloud computing is “What is the cloud?“. There are a lot of terms, vendor specific definitions and confusion about cloud infrastructure so we’ll first define cloud computing before moving on.

Solid Logic’s cloud computing definition: Instantly scalable, programmatically-controllable compute, storage, and networking resources. 

This definition is also commonly referred to as Infrastructure-As-A-Service (IaaS). Infrastructure-as-a-Service abstracts the physical aspects of IT Infrastructure and provides a set of application programming interfaces (APIs) to control all aspects of the infrastructure. It is very powerful and allows you to basically manage a data center from a development environment or software application.

Many of the people we speak to have never used Amazon Web Services (AWS), Rackspace Cloud or another IaaS cloud provider, for different reasons. We’ve used IaaS for everything from High-Performance Computing to Video Hosting to low-cost development/test or non-production infrastructure. Our experience serves as a guide on which workloads fit well within an IaaS structure and which ones do not. It also allows us to prescribe  a customized, phased approach to cloud integration that minimizes cost and business risk in different ways.

The next comment that normally comes up when speaking to people about cloud infrastructure is: “The cloud sounds great, I hear it saves a lot of money but its just too risky/insecure/complex for us.” 

Organizations that have not yet embraced IaaS or “the cloud”  in their business generally do so for similar reasons. Most of the reasons center around perceptions that may be outdated or untrue – it depends on their scenario.

In our experience, their reasons generally fall into one of the categories below:

• Cloud Performance (CPU, Disk, Network, Bandwidth, etc.) – I heard cloud servers are slow. The disks are slow and unpredictable. 
  
• Budgeting/cost modeling – How do I know or estimate what my costs will be?
  
• Cloud Security – It can’t be secure. Its called ‘Public Cloud’. Can other people access my files or servers?
• Cloud Reliability – Netflix went down so it’s not reliable. What do I do if it goes down?
• Cloud Compliance – No way, can’t do it – I’m subject to ABC, DEF or XYZ compliance requirements
• Cloud Audit requirements – No way, the auditors will never buy-in to this.
• Employee training – How do I find people to manage this?
• Steep learning curve – How do I get started? Its seems really complex.

Cloud misperceptions abound

As the saying goes, perception is reality, and there are also a lot of misconceptions that increase fear of the technology and prevent people from moving suitable workloads to the cloud.

Popular news sources perpetuate the myths about cloud computing. It seems that every time Amazon Web Services (AWS) (who is by far the largest cloud provider) has any sort of hiccup or downtime, reporters jump on the bandwagon that cloud infrastructure is useless and breaks too often. Here is a link to a Google news search for this: https://www.google.com/news?ncl=dvYSd5T83PVQigMPa1-2GMz-snaDM&q=aws+down&lr=English&hl=en

How we’re addressing these concerns

We’re going to address each of these concerns by sharing much of what we’ve learned along the way. We hope to shed some light on what seems to be an increasingly complicated market with more and more terminology and complex jargon used every day.

1. We’re working on a comprehensive cloud computing benchmarking report. The report will make an apples-to-apples comparison between cloud instance sizes and existing in-house infrastructure. It will use common benchmarking tests that anyone can replicate in their environment. It will allow organizations to make informed business decisions on whether or not they could benefit from integrating “the cloud” into their IT infrastructure and software development approach. Sign up here for a copy of the cloud computing benchmark report.
2. We’re going to present some cost models and budgets for common scenarios. We’ll integrate both tangible and non-tangible costs and benefits that we’ve searched for but haven’t seen included anywhere else. Contact us for a cost model for a specific use case.

Author information

Eric Detterman

Founder and CEO at Solid Logic Technology Inc.

Interested in Web Applications, Cloud Infrastructure, Quantitative Trading Systems, and more. Owner of an awesome St. Bernard named Barnaby.

TwitterGoogle+LinkedIn

Friction vs. Momentum in Life

This post is not focusing on the aspects of friction and momentum in a physics sense, but rather in a philosophical sense, and the way they impact a person’s business and personal life.

Author information

Eric Detterman

Founder and CEO at Solid Logic Technology Inc.

Interested in Web Applications, Cloud Infrastructure, Quantitative Trading Systems, and more. Owner of an awesome St. Bernard named Barnaby.

TwitterGoogle+LinkedIn

In the spirit of the upcoming election and debates, I thought it would be interesting to put out some something to summarize the words used by both of the candidates in the 2012 Second Presidential Debate on October 16, 2012. We grabbed the text from here. We’re not diving into anything overly complex here but it does put last night’s debate in a different context that we found interesting.

The way the graphic turned out is interesting: president, governor, jobs, thats people. 

Link to the WordCloud: http://www.solidlogic.com/wp-content/uploads/2012/10/wordcloud_debate_transcript.png

2012 Second Presidential Debate Word Cloud

Share this:   

How to build a word cloud

The easiest way to build a word cloud is to use one of the great free online tools like Wordle to build the graphic. If you need a more customized approach or need to create something like this in software, you can use several software tools to make it a lot easier. More details to come on the methods and code behind this later on but its based on Python and R, both of which we use quite a bit for data analysis and development projects.   The code for this was created by myself and our CIO, Michael Bommarito. Its based on some of the work he’s previously made available here: Wordcloud of the Arizona et al. v. United States opinion and Archiving Tweets with Python.

To get customized analysis like this, or to ask us anything else please use the contact us.

Author information

Eric Detterman

Founder and CEO at Solid Logic Technology Inc.

Interested in Web Applications, Cloud Infrastructure, Quantitative Trading Systems, and more. Owner of an awesome St. Bernard named Barnaby.

TwitterGoogle+LinkedIn

Solid Logic’s CEO, Eric Detterman, and CIO, Mike Bommarito, will be presenting at the AWS Michigan Meetup at Tech Brewery (map) in Ann Arbor, MI. I’ll be presenting on how we use Amazon Web Services (AWS) in the quantitative financial trading space with a case study and more.

Mike will be presenting on Legal Informatics using AWS CloudSearch. He will also be demonstrating an early prototype of an private enterprise information search and e-discovery application we’re creating. Mike also has a copy of his presentation available here.

Event Date: Tuesday, October 9th, 2012 @ 6:30pm

Event: AWS Michigan Meetup

More info: http://www.awsmichigan.org/events/85530922/

Below is a copy of my presentation so you can view at it at your convenience.

Author information

Eric Detterman

Founder and CEO at Solid Logic Technology Inc.

Interested in Web Applications, Cloud Infrastructure, Quantitative Trading Systems, and more. Owner of an awesome St. Bernard named Barnaby.

TwitterGoogle+LinkedIn

What happens if your problem doesn’t easily fit into the the Hadoop framework ?

Most of the work that we do in the financial sector falls into this category. It just doesn’t make sense to re-write existing code to fit into the Hadoop paradigm. Example case study here and blog post here.

As in any business, new ideas lose their ‘edge’ as they sit on the shelf or due to delays in the idea execution stage – primarily because of opportunity costs and increased chances of a competitor creating a product around the idea. The faster a concept can be brought to market, the larger the advantage to be had by the creator. This is especially true in the financial trading tech sector where advancements are measured in minutes/hours/days vs. weeks to months. Because of this, we’re always looking for new and creative ways to solve data and “big data” problems quicker.

Enter Apache Drill

One of the more interesting articles we came across recently focused on a new Apache project that aims to reduce the time to get answers out of a large data set. The project is named Apache Drill and here is a quick overview slide deck.

The Apache Drill project aims to create a tool similar to Google’s Dremel to facilitate faster queries across large datasets. Here is another take on the announcement from Wired. We’re excited about this because of the direct way this will impact our work and specifically the workloads that require real-time or near real-time answers.

Apache Drill Video Overview

Please use the form below to contact us to see if we can help solve your “Big Data” challenges:

[contact-form subject="SL Blog Response"] [contact-field label="Name" type="name" required="true" /] [contact-field label="Email" type="email" required="true" /] [contact-field label="Phone" type="text" /] [contact-field label="Comment" type="textarea" required="true" /] [/contact-form]

Author information

Eric Detterman

Founder and CEO at Solid Logic Technology Inc.

Interested in Web Applications, Cloud Infrastructure, Quantitative Trading Systems, and more. Owner of an awesome St. Bernard named Barnaby.

TwitterGoogle+LinkedIn

There are many different ways to combat these problems with varying levels of success. The is a huge industry of web security software and tools out there. The industry is changing rapidly due to changes in infrastructure because of cloud computing. The one approach that does not work well (and never has) is the set it and forget approach that many people use when they create a new site. Since website security is an on-going challenge, it’s best to use professional level services and stay up to date with everything. Unfortunately, some of these services can be quite pricey.

We take website security and performance very seriously and offer a range of services in these areas. We use multiple services and techniques to protect all of the public (and private) sites we create. One of the methods is a security service called CloudFlare which we will describe below and walk through a case study they published over the weekend.  Here is a quick overview of the service: CloudFlare is a quickly growing, venture capital backed web security and performance start-up.

CloudFlare presently serves over 65 BILLION (yes Billion, not Million) pageviews a month across their network of sites they support.

Here is some perspective on their size from a VentureBeat article: “We do more traffic than Amazon, Wikipedia, Twitter, Zynga, AOL, Apple, Bing, eBay, PayPal and Instagram combined,” chief executive Matthew Prince told VentureBeat. “We’re about half of a Facebook, and this month we’ll surpass Yahoo in terms of pageviews and unique visitors.”

They have a great list of features:

Over the weekend they had some interesting events happen in their European data centers and wrote a couple blog posts about it. Linked here and summarized below:

What Constitutes a Big DDoS?

A 65Gbps DDoS is a big attack, easily in the top 5% of the biggest attacks we see. The graph below shows the volume of the attack hitting our EU data centers (the green line represents inbound traffic). When an attack is 65Gbps that means every second 65 Gigabits of data is sent to our network. That’s the equivalent data volume of watching 3,400 HD TV channels all at the same time. It’s a ton of data. Most network connections are measured in 100Mbps, 1Gbps or 10Gbps so attacks like this would quickly saturate even a large Internet connection.

 

To launch a 65Gbps attack, you’d need a botnet with at least 65,000 compromised machines each capable of sending 1Mbps of upstream data. Given that many of these compromised computers are in the developing world where connections are slower, and many of the machines that make up part of a botnet may not be online at any given time, the actual size of the botnet necessary to launch that attack would likely need to be at least 10x that size.

 

In terms of stopping these attacks, CloudFlare uses a number of techniques. It starts with our network architecture. We use Anycast which means the response from a resolver, while targeting one particular IP address, will hit whatever data center is closest. This inherently dilutes the impact of an attack, distributing its effects across all 23 of our data centers. Given the hundreds of gigs of capacity we have across our network, even a big attack rarely saturates a connection.

 

At each of our facilities we take additional steps to protect ourselves. We know, for example, that we haven’t sent any DNS inquiries out from our network. We can therefore safely filter the responses from DNS resolvers. We can therefore drop the response packets at our routers or, in some cases, even upstream at one of our bandwidth providers. The result is that these types of attacks are relatively easily mitigated.

 

What was fun to watch was that while the customer under attack was being targeted by 65Gbps of traffic, not a single packet from that attack made it to their network or affected their operations. In fact, CloudFlare stopped the entire attack without the customer even knowing there was a problem. From the network graph you can see after about 30 minutes the attacker gave up. We think that’s pretty cool and, as we continue to expand our network, we’ll get even more resilient to attacks like this one.

Link to original post: http://blog.cloudflare.com/65gbps-ddos-no-problem

The big takeaway for us is that we’re in a better spot by using CloudFlare. There are very few security software tools or services out there that would be able to handle this sort of attack, mitigate it and then describe it in such a short period of time.

Author information

Eric Detterman

Founder and CEO at Solid Logic Technology Inc.

Interested in Web Applications, Cloud Infrastructure, Quantitative Trading Systems, and more. Owner of an awesome St. Bernard named Barnaby.

TwitterGoogle+LinkedIn

Event Date: Wednesday, September 12th, 2012

Event: Techonomy Detroit

More info: http://techonomy.com/ and http://conf.techonomy.com/12-detroit/overview

Author information

Eric Detterman

Founder and CEO at Solid Logic Technology Inc.

Interested in Web Applications, Cloud Infrastructure, Quantitative Trading Systems, and more. Owner of an awesome St. Bernard named Barnaby.

TwitterGoogle+LinkedIn

Software Development Life Cycle (SDLC)Importance

English: SDLC Phases Related to Management Controls (Photo credit: Wikipedia)

Solid Logic Technology’s foundershave experience across the financial industry and specifically in the development of quantative trading and investment systems. Many of the things we’ve learned along the way impact the way we develop software for clients across other industries. Most notably we’ve learned that software quality is extremely important and ‘software bugs’ cost lots of money. The study below shows how important in-depth software development, testing and launch management is for a company.

As early as August 1st, 2012 reports came out that Knight Capital Group,  a prominent electronic market-making firm specializing in NYSE equities, lost an estimated $440 million dollars due to a ‘software bug’. The news spread across financial news networks like Bloomberg, NY Times, CNBC and The Wall Street Journal.  Knight and other similar firms, trade US equities electronically using sophisticated computer algorithms with little to no human involvement in the process. While we will probably never hear the full story behind the ‘software bug’, it is suspected that a software coding error that was not quickly identified caused the loss. The loss is approximately 4 times their 2011 net income of $115m. It appears to have pretty much decimated the firm and at this point it looks like the firm will be bought or end up in bankruptcy.

While unfortunate, this example has some implications across any software project.

So what can we learn and take away from this incident?

1. Software is not perfect, especially right after it is released
2. A more comprehensive Software Development Life Cycle (SDLC) process and launch plan probably would have reduced the loss to a more reasonable amount.
3. Always have a contingency plan for a new launch
4. If new software is ‘acting funny’ then it probably has a problem and needs to be pulled from production and fixed
5. When possible, conduct a series of small ‘pilots’ or ‘beta’ test along the way in a lower impact way
6. If you cannot fully test the changes, then implement them slowly to minimize the potential errors in the beginning
7. Have a ‘kill switch’ and know how to use it
8. Have a formal SDLC process and follow it for all revisions
9. Use source control for all software changes
10. Have a defined launch process
11. Have a way to quickly revert the changes implemented back to the previous version.

These are basic best practices that all software development firms should follow in order to consistently develop high quality software. Its unfortunate that there is a case study like this but these type of incidents are more common (but not to this scale) than most people imagine. I’m sure the group at Knight completed many of the above items, but something got away from them.

We put a huge amount of thought and effort into the process of software development and the consistent high level of quality that a solid process brings. We’re currently working on publishing a set of Software Development best practices – please contact us for a pre-release version.

More posts by Eric

Share this:   

Related articles

• Software development
• Systems Development Life Cycle (SDLC)

Author information

Eric Detterman

Founder and CEO at Solid Logic Technology Inc.

Interested in Web Applications, Cloud Infrastructure, Quantitative Trading Systems, and more. Owner of an awesome St. Bernard named Barnaby.

TwitterGoogle+LinkedIn