How to handle non transient exception in Azure Worker role -

we have 2 azure worker roles - , b.

• a quartz scheduler runs jobs every minute.
• it reads ids 'redis cache' every minutes , execute jobs ids.
• 'a' publish output service bus queue subscribed worker role 'b'.
• 'b' worker role reads values queue , execute more operation on them.
• both worker roles has build cache on startup.

now here few issues regarding azure component failure:

• if redis cache goes down, how can handle that. need stop our execution till time again , need build our cache again. 'b' worker role should stop pulling message service bus till time redis comes again.

• how handle service bus failure in worker role 'b'?

you don't need stop of worker roles.

worker role should resilient issues in redis cache, meaning code should handle exception thrown redis (or network exceptions), either retry or swallow exception.

worker role b should pull messages service bus. if worker role doesn't publish data, worker role b should handle empty results.

stopping service on redis/azure glitch require handle more complicated scenarios - example: automatically detect if redis again , automatically start service.