This article diagnoses a complex problem and offers time saving tips. To skip the tips and diagnosis, and go straight to a simpler solution, click here.
Apache Camel is an easy and fast way to do otherwise hard tasks – why can it take so long for some of us to learn it deeply? It doesn’t have to.
A random list of Camel learning challenges:
- Welcome to Camel, where every dot can be a new research project!
- With so many ways to do anything, first, use up all your time budget just researching which approach to use!
- Type converters, generics, and overloaded methods are your friend. Except when you are learning.
- It’s only faster to code than native xyz code, when you also know enough of xyz to know what it’s faster than, so you can configure camel properly.
- Snippets help the documenter, perhaps not as much the learner.
- Fuse, fabric, other deployment/run options are super! But have almost nothing to do with Camel except by association, and as consumers and servers. Confusing.
- It’s normal to read docs, instead of the tests in the source code.
- To learn anything, you need real data to learn with. Another distraction: You need to learn how to acquire the data, first.
- The need for realistic examples that you can relate to (broad) is always in direct conflict with realistic examples that show you how to do things (narrow).
- Project setup and deployment is a separate obstacle, and a distraction for learning how the fluent API works.
- Straight Math: (Number of possible components) X (number of configuration options for each) X (number of ways to wire them together) = (brute force memorization barely even an option).
- There appear to be no consistent conventions. This is an illusion, but a really convincing illusion. Everything you learn with one, may seem completely different, with another endpoint or component. Maddening.
- Version congruence: This can waste much of your time. Know why.
Look, as a developer, none of the above even matters, until it starts wasting your time. Then, it matters a lot. Above all, please respect my time. Here’s what I learned about how to reduce wasted time, with Camel.
Is Learning Instinctual?
Camel, and Cognitive Load
Scientists sometimes observe correlations between intelligence and a tolerance for ambiguity. This article will attempt to harvest your own best capabilities for learning Camel, by identifying those areas where cognitive load feels like it might blow your circuits.
Where this gets interesting is when you inject time into the learning model. Put an oversized Cognitive Load into the model, and time increases way out of scale. This becomes real when you only have a weekend to study something, and a Cognitive Load entirely prevents you from learning that one “xyz” task that you set out to master. Failure vs success, with the next available weekend coming up in 7 weeks.
Anticipate properly, and you can absorb patterns that might otherwise seem bizarre, especially to those of us who prefer a more deterministic feeling approach than Camel sometimes provides.
Sal Khan, founder of Khan Academy writes beautifully about the process of learning, especially as it relates to the importance of gaining context first, and sequentially. From that perspective, the simple task of identifying landmines to the learning process can assist us the learner in getting those out of the way, before they frustrate you.
Acknowledging Claus Isben
Perhaps your best, and worst, friend is Claus Isben
- Seems to have written Apache Camel, almost single-handedly (? true?)
- Has written the primary go-to book “Camel in Action”
- Has written most and best answers on Stackoverflow
- Has been a best presenter, wherever he goes, including youtube.
Claus suffers from the same handicap that anyone who has already learned Camel suffers from. It’s hard to remember which foundational understanding is taken for granted in any teaching context. So his best efforts might sometimes assume a necessary understanding which you lack. This is not avoidable, not by Claus, nor by any human.
Definition: “Learning Camel”
“Learning Camel” in this article should means a depth beyond the ability to run working projects. It’s easy to use Camel even without learning it to this depth – and I have done so.
But when it’s time to really learn Camel deeply, here is what that could mean:
- Enough Camel skills to pass the EX421 Certification Exam
- Enough Camel skills to be fast when writing routes, setups, etc.
- Enough Camel skills to be authoritative as an architect or designer, knowing how and when to use Camel in every situation.
- Enough Camel to pass an interview well.
- DOES NOT include edge cases, such as how every single obscure component might work. That can be learned as needed.
“xyz” in this article is defined as whatever it is that you are attempting to learn at one given moment. A component, an endpoint, a transformation, configuration, or … whatever that is, “xyz” means something as narrow as possible, lest you get lost in the sauce.
I don’t usually have to tool up so heavily to learn a new skillset. But most skillsets aren’t like Camel. These were in my toolkit:
- The book: Claus Isben’s Camel In Action is one great choice.
- [optional] The code from the book. Perhaps even more helpful than the book itself, at times. See below.
- The entire Camel codebase ($git clone …). More below, but not optional.
- Workspace usage and configuration. See below. This is can slow you down if you approach it the normal way.
- The Camel site, and Stackoverflow.
Strategy: Bite Size
With Camel, you can eventually do anything. Repeating for emphasis, the trick to is learning it faster is the sequence that you learn it in.
- before you read the book.
- before you run your first examples.
- before you commit to a course of action.
Before you do anything, learn the trip-wires that can keep your brain spinning needlessly. Once you do that, you can can learn sequentially, and it may be a much faster process.
Land Mines, Trip Wires
There are at least four ways that Camel designs it’s API to be any-to-any, and they all have the side effect of obfuscation or confusion. Candidates for Cognitive Overwhelm. These include.
- Type Converters
- Generics and overloaded methods
Sure, you can often do a lot of Camel without worrying about these trip-wires and land-mines. But that’s not the same thing as really learning Camel. See Definition: Learning Camel above.
Defense against Type Converters
Every time you type a dot (.) in the Camel fluent API, you may be engaging a helpful Type Converter, acting in your behalf, but without your knowledge.
WTF? “I thought that the message was a [….] at this point.” But no, it’s a [….] and Camel just did this conversion to help you. The fact that it doesn’t feel so helpful when you have no idea what is going on under the covers is the point here. You have to get used to this, and usually the only defense is to engage in some form of forensics to learn what Camel did “for you.”
Such forensics are beyond the scope of this article, but one key defense is below, in red.
Defense against Generics and Overloaded Methods
Code-completion in the IDE gives you the idea that the fluent Camel route API uses static typing to guide you along your way. Cool!
But wait! It specifies most arguments using java generics to let you pass all manner of good stuff around. It’s statically typed, sure, but then you learn that all those static typed arguments are little more helpful than unfamiliar marker interfaces! How helpful does that feel? It often feels impossible (to me anyway) to figure out exactly what and how I am expected to be passing arguments into the next part of the fluent API. This, combined with overloaded methods, and a combination of 6 ways to do anything… one unfamiliar fluent API call can take you a whole morning to research, especially if you look in the wrong places first, which I have done a lot of.
Reminder: All this functionality is only here to help, even if it doesn’t feel that way when you are scratching your head in cognitive overwhelm.
As above with Type Converters, it takes forensics to figure this stuff out. One primary defense is below, in red.
Defense Against Configuration Challenges
Routes and endpoints change often, duh. OTOH, project and component configuration remain relatively static, especially on a single team, but they are only static once you achieve them! Learning Camel means that instead of learning about routes and endpoints, instead you are learning about xyz configuration, and project configuration, and that can take a some time.
There are many primary defenses against the challenges of learning project and xyz configurations.
- Learn configuration separately from the routes and endpoints, if at all possible.
- Start with example projects that are as narrow and close to the configuration you need, as you can get.
- See list below on what to “Avoid These As Long As Possible”
Defense against Multi-Everything
I can offer so many examples to illustrate the scope of this multi-everything problem, I just have to pick one:
In prepping for the exam, I was attempting to build an exhaustive list of splitting situations so I could knock any one that came up in 15 minutes. So of course, one of the splitting use cases involves xml. But starting the research I instantly came upon not one, but many ways to split xml. Too many to go over here. This research alone involved many different sources including grepping camel source code, and several books and reference docs.
So I built the capability on xpath, since that’s where most of the examples I found were. Then days later, while building examples of string splits, I happened upon yet another source of information deep within the camel site. There I learned about tokenizeXML and xtokenize built for special use cases where xpath doesn’t work as well. So now it’s back to the drawing board, and I am building 3 different sets of examples for splitting xml, not one. Total time invested is too embarrassing to reveal here.
I offer no definitive approach to resolving this TMI nightmare, but I do offer some heuristics.
- Unprofessional? I literally have to coach myself through the unprofessional feeling I get, when I can’t investigate every single approach first, before committing to a course of action. Yes, I’d really like to know which one works the best and in which circumstances, but that would take too much time, in Camel land. 🙁 Give yourself permission to not learn everything, and hope and pray that the approaches you decide to not learn aren’t on the exam, or in an interview.
- Occam’s Razor: Stick with the simplest approach first, or the first that you find first, or the one that you get working first. You will probably find a better way later 🙁
- Quick Surveys and Memorization: Sometimes I can coach myself with word-tracking that goes like this. “Well, the book gives me these 6 different approaches to writing a transform, but I don’t have time to learn them all in detail right now. So maybe just memorizing the names of these 6 approaches would help me in an interview.”
If you love to code, and you contract out as an architect, and you have an obsessive personality, saying no to all these super great possibilities may be the hardest part about learning Camel. And the most important.
Runnable Code is Preferable to Documentation
A helpful strategy is to opt for running example code over documentation, or at least, sequentially. Documentation is invaluable, but usually makes much more sense after getting an example running.
Especially important is sparse running code. The exact minimum is the best, especially in Camel where it’s so easy to go off on tangential concerns such as upstream or downstream endpoints.
I believe in documentation, so if you’re having a hard time with the idea of ignoring documentation, the way I get my head around it is to imagine how insanely difficult it would be to write adequate documentation for something as flexible as a Camel endpoint or component. One component could be an entire chapter, or even book, if you were meticulously thorough about all the incoming and outgoing options, with examples. Ridiculous.
Budget Time for Project Setups and Data Feeds Separately
An old joke about Regular Expressions: “There was this guy who had a programming problem that required a Regular Expression. Now he has TWO programming problems.”
May be odd to say this, but Camel project setup requires a different kind of mental state, when in cognitive overwhelm. Get that out of the way before thinking about your routes much, or trying to learn “xyz.”
And setting up your data feeds too. Get that done before trying to learn how to do xyz with the data. One more distraction out of the way. At this point, you’ve got your data ready to process, all you have to do is xyz…
The nice thing about getting your setups done separately is you can often reuse such setup projects.
I keep a “shared” project as a dependency. It’s got enough of the basic data feeds and configuration dependencies to give me most of what I need. Then I set up a separate xyz project just for the delta – that part that corresponds only to xyz.
10 Minute Reps
Cognitive Load is a fancy name for whatever you actually have to think about. So if you need to focus on learning xyz, everything before needs to already be a part of your muscle memory.
Doing reps on the setups is one approach to dumping that portion of the learning into your muscle memory.
Muscle memory in Camel is easier than you might first guess, with another reps. Jammazwan.setups is designed for gaining such muscle memory.
Avoid These, For As Long As Possible
When possible, there are super excellent aspects of Camel that you should just plain avoid, until you are out of the basic learning phase. Learn Camel first.
- Deployment options.
- Expression language options. (Go with default).
- XML routing options, blueprint, spring xml, etc.
- Arcane components such as Avro or Undertow or …
It’s not that you shouldn’t learn them, just don’t try to learn them at the same time as you are picking up xyz, or configuration basics, or …
Your strategy here might be compared with a child eating dessert. You have a delicious treat waiting for you, after you eat the vegetables.
Accommodation: Meaningful Example Projects
In direct opposition to the need for sparse, or narrow, sample projects (see above) there is a need to keep your brain interested and engaged. Sparse projects can be very dull. It can be hard to relate to “Hello World”.
To do this, you may naturally make the project more realistic. This is especially true if you are ADHD (like me) and you can only concentrate for long periods of time, on stuff that is at least mildly interesting.
The strategy here is to go ahead and make it interesting if you must, but handicap yourself, attitude wise. Explain to yourself that you’re taking some chances by adding breadth to the project that might be a bit distracting.
Many times, it turned out that I spent only half hour on learning xyz which was the point of an exercise, and a many times that on ancillary issues that were really only designed to make the project meaningful, such as realistic data inputs, or downstream distribution that felt meaningful.
An example would be “This project would feel much more real to me if it was like our other projects, which always FTP the results to the abc server.”
So instead of spending your time budget learning the syntax/semantics for xyz, you are configuring a dev instance of abc server, setting up permissions, etc etc.
How Not To Be Sidelined By Version Issues
Everywhere you look on the Camel site, you will see these odd references to what version which feature is deployed into. These are much more prominent than what you might see on other open source projects. Why such a prominent display?
Turns out that there is a very real reason, and I burned up some time trying to diagnose odd errors that were only caused by version differences. How could this be? Just use the latest, right? Not so fast …
The Camel codebase moves much faster than, for example:
- Your deployment options. You may have to deploy to a much older version of Camel or Fuse or Karaf or EAP at work.
- Your testing options. If you study on the latest, understand that the Red Hat Exam which costs hundreds of dollars might be written on a version of the API that doesn’t have lots of the features you are learning!
- Tutorials, docs, sample code versions. Lots of docs, tutorials, and sample projects were written years ago. Helpful to have, except whoops Camel has moved on so what you are reading may not be relevant.
- Your tooling options: Red Hat’s IDE comes to mind.
- Camel DOES NOT MIX WELL with other versions of itself. Don’t expect to use one camel version of one dependency, and another camel version of another dependency. You have to move them all in lockstep. You don’t want to find out the hard way what happens if you try to mix them.
Knowing what you are up against helps you plan your learning around these obstacles.
From above, I assume you have
- An IDE such as a version of Eclipse
- The camel code.
- book code [optional]
- Multiple workspaces, as detailed below.
There are important workspace segregation strategies that I find helpful when learning Camel, one is a life-saver.
- Keep your the Camel and Camel In Action codebases out of your working workspace. It’s too much code, and will suck up all your IDE’s memory before it even fully opens. Especially, give the Camel code it’s own workspace. Even then you’ll probably have to give your IDE extra memory.
- Maintain a separate workspace for Camel code only. Build with maven so you have fully compiled code in view. Only import Camel modules that are relevant. Only keep open those camel modules that you really need to search in (usually only camel-core or the camel-xyz component that you are learning.)
- At the first moment of cognitive overwhelm on any xyz topic, reflexively grep for “.json(” in *Test.java” files first, where json is the xyz that you are attempting to learn. This one tip may save you quadzillions of hours. Search in both the book code, and in camel’s own source code. I use Eclipse file search (control H) for this grep, your approach may differ.
- Be ready to use grep in the bash shell or comparable search to above in your favorite text editor such as sublime, for times when opening the IDE in the Camel workspace isn’t worth the trouble.
- If you opt to maintain the Camel In Action book’s code as an additional resource – recommended – the same rules apply for it as for the camel code above. Separate workspace, fully compiled, grep as above…
Fan-Out is Actually a Different XYZ
I saved this for last, because it might be a bit of a head scratcher at first.
One thing I run into a lot has to do with fan-out.
- I am learning xyz
- I got it down, I think – my sparse project is working
- Feeling confident, I move on a new xyz
- Two days later, I discover that I didn’t have the previous one down like I thought… WTF?
The simplest use case I can think of is json. But you can apply this to many similar situations.
I found that going back and forth between json and other inputs/outputs was super easy, and fast to learn. But I only did so with single inputs, as per the rule above, of using sparse example code. Mission accomplished!
But the first time I actually tried to consume some real json, however, I try to injest a file with multiple json records, and I’m lost. For this, I had to learn to incorporate jsonpath, and splitter, and aggregation. Which in turn required a bit of research on the actual expression for jsonpath that had nothing to do with Camel. Hmmm.
So I had learned json marshalling and unmarshalling, but in another respect I really hadn’t. Because the fan-out to multiple records is a totally separate problem space. Welcome to your brave new world.
Fast Supporting Setups
Docker isn’t the only way, but however you achieve fast setups for your supporting infrastructure, this can be a good thing.
Here’s how I did mine..
When learning other skillsets this kind of shortcut might not be necessary. With Camel, every 20 hours you save might make the difference between giving up before you finish, or not.
Learning Apache Camel can take a long time, but there are strategies that can help you shortcut the process.
This article diagnosed a problem. To review one solution, click here.