java - Understanding Apache Spark filter transformation behavior -

-

i have list of items in javardd, every item date (java calendar). now, want filter dates less given date. that's code:

main

    public static void main(string[] args) {         sparkconf conf = new sparkconf().setappname("date comparison test")             .setmaster("local[4]").set("spark.executor.memory", "1g");         javasparkcontext sc = new javasparkcontext(conf);          // initializes filter date 01/01/2016 @ 10:00:00         calendar filterdate = calendar.getinstace();         filterdate.clear();         filterdate.settimeinmillis(1451642400000l);          // initializes array of 40 calendars, in every date         // 1 hour later previous, starting         // 01/01/2016 @ 08:00:00         arraylist<calendar> calendararray = new arraylist<>();         // milliseconds corresponding 01/01/2016 @ 08:00:00         long initial = 1451635200000l;         for(int i=0; < 40; ++i) {             calendar 1 = calendar.getinstace();             one.clear();             one.settimeinmillis(initial);             calendararray.add(one);             initial += 3600000;         }          javardd<calendar> rdd = sc.parallelize(calendararray);         javardd<calendar> rddfiltered = rdd.filter(new filtertest(filterdate));         system.out.println("rdd size " + rddfiltered.count());         sc.close(); }

filtertest code

public class filtertest implements function<calendar, boolean> {  private static final long serialversionuid = -3134317182912968444l; private final calendar filteringdate;  public filtertest_(calendar filteringdate) {     super();     this.filteringdate = filteringdate; }  @override public boolean call(calendar arg0) throws exception {     // getstandardformatteddate prints date in given format     system.out.println(timeutils.getstandardformatteddate(arg0) + " - " + timeutils.getstandardformatteddate(filteringdate));     if(arg0.before(filteringdate)) {         return false;     }     else {          return true;         }   } }

what can't understand output get. seems fixed calendar pass parameter compare changes (like when it's sat, 01 jan 2016 22:00:00).

output

sat, 01 jan 2016 08:00:00 - fri, 01 jan 2016 10:00:00 sat, 01 jan 2016 08:00:00 - fri, 01 jan 2016 10:00:00 fri, 01 jan 2016 08:00:00 - fri, 01 jan 2016 10:00:00 sat, 02 jan 2016 15:00:00 - fri, 01 jan 2016 09:00:00 fri, 01 jan 2016 09:00:00 - fri, 01 jan 2016 10:00:00 sat, 02 jan 2016 15:00:00 - fri, 01 jan 2016 09:00:00 fri, 01 jan 2016 10:00:00 - fri, 01 jan 2016 10:00:00 sat, 02 jan 2016 20:00:00 - fri, 01 jan 2016 10:00:00 fri, 01 jan 2016 10:00:00 - fri, 01 jan 2016 10:00:00 fri, 01 jan 2016 10:00:00 - fri, 01 jan 2016 10:00:00 fri, 01 jan 2016 10:00:00 - fri, 02 jan 2016 07:00:00 sat, 02 jan 2016 17:00:00 - fri, 01 jan 2016 10:00:00 fri, 01 jan 2016 11:00:00 - fri, 01 jan 2016 10:00:00 sat, 02 jan 2016 21:00:00 - fri, 01 jan 2016 10:00:00 fri, 01 jan 2016 10:00:00 - sat, 01 jan 2016 22:00:00 fri, 01 jan 2016 22:00:00 - fri, 01 jan 2016 10:00:00 sat, 01 jan 2016 22:00:00 - fri, 01 jan 2016 12:00:00 fri, 01 jan 2016 23:00:00 - fri, 01 jan 2016 10:00:00 fri, 01 jan 2016 23:00:00 - fri, 01 jan 2016 10:00:00 fri, 01 jan 2016 23:00:00 - fri, 01 jan 2016 10:00:00 fri, 01 jan 2016 10:00:00 - fri, 02 jan 2016 00:00:00 sat, 02 jan 2016 00:00:00 - fri, 01 jan 2016 19:00:00 fri, 01 jan 2016 13:00:00 - fri, 02 jan 2016 10:00:00 sat, 01 jan 2016 10:00:00 - sat, 01 jan 2016 14:00:00 sat, 02 jan 2016 01:00:00 - fri, 01 jan 2016 10:00:00 fri, 01 jan 2016 10:00:00 - fri, 02 jan 2016 11:00:00 sat, 01 jan 2016 14:00:00 - fri, 02 jan 2016 11:00:00 sat, 02 jan 2016 11:00:00 - fri, 01 jan 2016 15:00:00 fri, 01 jan 2016 15:00:00 - sat, 02 jan 2016 10:00:00 sat, 02 jan 2016 02:00:00 - fri, 01 jan 2016 10:00:00 sat, 02 jan 2016 15:00:00 - sat, 02 jan 2016 10:00:00 fri, 01 jan 2016 10:00:00 - fri, 01 jan 2016 10:00:00 fri, 01 jan 2016 10:00:00 - fri, 01 jan 2016 10:00:00 fri, 01 jan 2016 12:00:00 - fri, 01 jan 2016 22:00:00 sat, 01 jan 2016 17:00:00 - fri, 02 jan 2016 10:00:00 fri, 01 jan 2016 22:00:00 - fri, 01 jan 2016 10:00:00 sat, 02 jan 2016 13:00:00 - fri, 01 jan 2016 10:00:00 sat, 01 jan 2016 10:00:00 - fri, 01 jan 2016 10:00:00 sat, 02 jan 2016 23:00:00 - fri, 01 jan 2016 10:00:00

what happening during computation distribution variable? because apparently result correct, i'm having trouble in debugging code in more complex situation.


Comments

Post a Comment

Popular posts from this blog

sequelize.js - Sequelize group by with association includes id -

-
so when requesting group sequelize follows: return models.workingcalendar .findall({ attributes: [ 'workingcalendar.periodid', 'workingcalendar.date', 'period.name' ], include: [ { model: models.period, attributes: [] } ], where: { getsudoid: currentgetsudo.id, unitplantid: unitplantid }, group: ['workingcalendar.periodid', 'workingcalendar.date', 'period.name'], }); sequelize run query: select [workingcalendar].[id], [workingcalendar].[periodid], [workingcalendar].[date], [period].[name] [workingcalendars] [workingcalendar] left outer join [periods] [period] on [wor...
Read more

java - Android raising EPERM (Operation not permitted) when attempting to send UDP packet after network connection -

-
when responding supplicant_state_changed_action broadcast , attempting send udp packet, exception: java.net.socketexception: sendto failed: eperm (operation not permitted) @ libcore.io.iobridge.maybethrowaftersendto(iobridge.java:542) @ libcore.io.iobridge.sendto(iobridge.java:511) @ java.net.plaindatagramsocketimpl.send(plaindatagramsocketimpl.java:184) @ java.net.datagramsocket.send(datagramsocket.java:305) @ com.mycompany.app.serviceclass.discover(serviceclass.java:355)at com.mycompany.app.serviceclass.access$600(serviceclass.java:39) @ com.mycompany.app.serviceclass$statemachine.run(serviceclass.java:242) @ java.lang.thread.run(thread.java:818) caused by: android.system.errnoexception: sendto failed: eperm (operation not permitted) @ libcore.io.posix.sendtobytes(native method) @ libcore.io.posix.sendto(posix.java:211) @ libcore.io.blockguardos.sendto(blockguardos.java:278) @ libcore.io.iobridge.sendto(iobridge.java:509) @ java.net.plaindat...
Read more

c++ - Migration from QScriptEngine to QJSEngine -

-
i'm migrating qscriptengine code on qjsengine. now, have: class pars { public: static qscriptvalue printmainlog(qscriptcontext* c, qscriptengine* e); }; qscriptvalue pars::printmainlog(qscriptcontext* c, qscriptengine* e) { //some actions return e->globalobject().property(""); } ... qscriptengine engine; ... engine.globalobject().setproperty("printlog",engine.newfunction(pars::printmainlog)); so, user can put printlog("what ever"); in application in , example, qlineedit , function pars::printmainlog evalute. is there way qjsengine? so, user put same printlog("what ever");? way find here , user should put logger.printlog("what ever"); logger class inherited qobject printlog slot.
Read more