- Protocols »
- API clients »
- Cloud providers »
- Throughput »
- Platforms supported »
- How it works »
- Getting started »
Protocols
What protocols does APILoader support ?
Yours. Among others.
We upload your API client. That talks directly to your server under APILoader's control. APILoader doesn't send server requests so it doesn't need to understand your protocol.
You do have to tell APILoader the name (any name) of your protocol. But only so it can match load generators to servers.
Does APILoader support Http/Https ?
Yes.
Http and Https are no different from other protocols as far as APILoader is concerned. Your APIClient implements the protocol. APILoader doesn't need to know.
(Actually APILoader does support recording and playback of Http and Https conversations but support for this likely to be dropped.
API clients
What is an API client ?
An API client is a lump of code that turns a call to it into a request that is sends to the server. It receives the response from the server and returns to the caller one or more objects that represent the response. The API client lives on the client machine (hence the name). Typically the API client is provided by the owner of the servers. An example is the Amazon Web Services (AWS) EC2Client. AWS customers embed this client into their applications through which they can then allocate and control AWS cloud machines.
What language API clients does APILoader support ?
Java.
Do you plan to support other languages ?
No.
But if you need another language supported we would be prepared to look at it as custom development. However the language would have to support reflection i.e. the ability of code to look inside a class and discover its constructors and methods. That precludes C-based languages other than C#.
What if I don't have a Java API client ?
To determine if APILoader is right for you it may be worth writing, or having us write, a Java API client that supports just the API calls used in the proof-of-concept use case.
Target applications
Can I test a stateful / stateless application ?
Yes. The principal difference is that when testing a stateful application proper attention to think time is required as think time affects the amount of time for which state is held in the server and therefore server resource requirements. We provide for think time in the construction of use cases.
Can I test a mobile application ?
Assuming you have encapsulated the mechanics of conversation between the mobile app into one or more lumps of Java code (better known as server API clients) then Yes. The lumps of Java code are no different from other API clients except that they probably use different libraries. As with any Java API client that you upload, you upload any necessary supporting libraries with it.
Cloud providers
What cloud providers does APILoader support ?
Currently we support Amazon Web Services.
Can you support other cloud providers ?
Yes, provided that the cloud provider has a Java API client that supports:
- allocation and deallocation of Linux cloud machines
- remote ssh access to the allocated machines
- ideally, access to some form of cloud storage
However, providing that support would be custom development billed at standard development rates.
Do you plan to implement support for other cloud providers ?
Yes. But not anytime soon. There are things to be done that will have more impact on the product's attractiveness given that we already support running in the cloud and the actual provider doesn't really matter to APILoader.
If you need a particular cloud provider supported we can bring that forward in the road map but it would be billed as custom development.
Throughput
Can I expect to achieve the throughput you quote ?
Probably.
The first issue is the response time of your servers. The longer that response time the larger the number of threads (aka virtual users) required to achieve a given level of throughput. We don't yet support millions of threads so we can't achieve the quoted top end rate on servers with response times in seconds. But if your server response time is below 100ms you should be able to obtain the quoted throughput.
The second issue is think time. Think time has to be added to server response time in determining the number of threads necessary to achieve a given level of throughput. That having been said we would not expect servers operating at the quoted level of throughput to be supporting clients directly - they are much more likely to be driven by other machines responding to external events. As such it is less likely that think time will be relevant.
Can the current throughput high-water mark be raised ?
Probably.
The limitation at the moment is a back-end component - the one that receives timers and stores them in the data mart. This is currently unable to scale to the larger AWS machine types. No doubt this can be overcome. If it is then linear scaling to the top end of the AWS machine range would suggest a throughput around 10M API calls per second. Naturally, the scaling won't be linear and another bottleneck will arise. However we knocked over bottleneck after bottleneck to get to the current high-water mark (2,000,000 API calls per second) and can't see why that process can't be continued. (We are, after all, performance engineers !) The only reason we haven't pursued this is that we feel that throughput is high enough for the moment and that there are other things that will make APILoader more useful.
Should we be unable to scale the above-mentioned component to larger AWS machines there is a second option. That is to replicate the component. That seems quite doable. But we thought it would make sense to improve its scalability before replicating it which is why we haven't done it.
Do you plan to raise the current high-water mark throuhput ?
Yes. But not anytime soon.
If you need the high-water throughput raised we can bring that forward in the road map but it would be billed as custom development.
Test types
What control do I have over the load in a run ?
A test run is based on a test run template. The degree of control you have over the run varies with the options supported by the chosen template.
Currently there is a single template. It allows you to escalate and de-escalate the load freely over the course of the run. You break the load into periods. In each period you can set a specific level of load to be run or you can allow APILoader to run the use case as fast as it can. The run length is the sum of the period lengths.
Other templates are possible but each will need a supporting screen to be developed so that the options of the template can be specified. So far we simply haven't seen the need.
Platforms supported
What platforms does APILoader run on ?
Linux.
Do you plan to support other platforms ?
No. We might be wrong but we don't see how it would matter.
How it works
How does APILoader know what methods my API client provides ?
Reflection. We instantiate your API client and look inside it with Java reflection. (Which is why we require your API client to be in Java and why we have to put a wrapper round it.) This allows us to discover the public methods and constructors that the client supports.
How do I provide run-time data ?
Generally API client method calls take parameters. Their types are typically a mixture of standard Java types and API-client-defined types. When APILoader asks you to provide a value for an API client method call it provides access to ways to provide values for standard Java types (strings, integers, etc). It also provides access to the constructors for the API-client-specific types. The constructor will require parameters. As for the method call these will be a mixture of standard Java types and API-client-specific types. For the standard Java types we provide access as above. For the API- client-specific types we provide access to the necessary constructors. And so on recursively until the method parameters are fully specified by standard Java type values.
At run time APILoader will invoke the constructors recursively until the API-client method call is fully specified. Then it will invoke the method call and load the server.
What facilities are there for providing values ?
For API-client-specific types we provide access to the constructor. Where the constructor parameters is another API-client-specific type we provide access to that constructor. Eventually, after enough levels, the method parameters will be fully specified by reference to standard Java types.
For standard Java types we provide:
- constants
- transaction scope variables
- tool-specified run-time parameters
- unique name / value generators
- output of earlier calls
- run-time data files
- Java code
How do I use Java code to provide run-time data values ?
The wrapper that we put around your API client allows it to be introduced to APILoader. But it also provides a raft of facilities. These include a data generator class which you can expand. All public methods of this class become visible, via Reflection, to APILoader and can be invoked to provide run-time data values.
This facility allows you to get your run-time data from pretty much any place you like in any manner you like.
We also use this facility to provide utility methods - such as converting a single instance to a list, taking the first item in a list, and so on. You can extend the list of utility methods as well.
Getting started
How do I get started ?
First you need to contact us. And we need to take you on as a customer. (See our business model.)
Then we need to code a simple Java wrapper round your Java API client.
But you said there was no programming involved in testing my API client-based system.
We lied !. Actually we didn't. We said there wasn't any programming for you. Which is true. We will write the wrapper. Its one-time only and pretty simple. But its easier for us to do it than document the process.
Once that is done we are able to run APILoader on your behalf. You will need to arrange access to your system-under-test. Which may require us to be on site.